Image processing device and method, endoscope system, and program

ABSTRACT

The present technology relates to an image processing device, an image processing method, an endoscope system, and a program that enable motion detection with higher accuracy. The endoscope system includes: a light source capable of performing a pulsed-light emission; a light-source control unit that controls the light source such that the light source performs the pulsed-light emission a plurality of times in an exposure time period of each captured image; an imaging unit that takes the captured images; and a motion detection unit that detects a motion of a photographic subject in the captured images. The present technology is applicable to the endoscope system.

TECHNICAL FIELD

The present technology relates to an image processing device, an imageprocessing method, an endoscope system, and a program. Moreparticularly, the present technology relates to an image processingdevice, an image processing method, an endoscope system, and a programthat enable motion detection with higher accuracy.

BACKGROUND ART

Hitherto, in an endoscope device, in order that observation is performedwith higher accuracy, motion detection may be used in combination withimaging of a photographic subject.

As an example of technologies that use the motion detection, there maybe mentioned a technology including detecting motions of thephotographic subject and a camera, and utilizing these motions for imagestabilization. As another example, there may be mentioned a technologyincluding detecting periodical motions of the photographic subject, suchas pulsation, acquiring images in-phase with each other on the basis ofresults of the detection, and outputting and displaying unblurredin-phase images. Further, as still another example, a technologyincluding detecting the motion of the photographic subject, and applyinga result of the detection to scene recognition has also been proposed.

As a technology for performing such motion detection, a method includingdetecting a motion from a plurality of previous frames, and imaging anddisplaying a result of the detection has been proposed (refer, forexample, to Patent Literature 1).

CITATION LIST Patent Literature

Patent Literature 1: Japanese Patent Application Laid-open No.2011-172783

DISCLOSURE OF INVENTION Technical Problem

However, in the above-described technology, there are difficulties indetecting the motion with high accuracy.

For example, in the method including using images corresponding to theplurality of frames, information items of two or more previous framesare needed to calculate the motion. Thus, a motion information item tobe obtained is calculated from relatively old information items such asthat of a penultimate frame. Therefore, a motion in a current framecannot be estimated with sufficient accuracy. In other words, at a timeof using the motion information item for image correction processes suchas the image stabilization, a mismatch occurs between a detected motionand an actual motion in the current motion to which the result of themotion detection is applied.

Further, in the method including using the images corresponding to theplurality of frames, the motion detection cannot be performed withrespect to motions that fluctuate at a rate higher than frame intervals,and to vibrating motions in short periods. In addition, even in a casewhere a motion between frames can be detected, when the motion is rapid,motion blurring occurs in images. Thus, inter-frame image mismatchesoccur to decrease accuracy in motion detection.

The present technology has been made in view of such circumstances so asto enable motion detection with higher accuracy.

Solution to Problem

According to a first aspect of the present technology, there is providedan image processing device including:

a light-source control unit that controls a light source such that thelight source performs a pulsed-light emission a plurality of times in anexposure time period of each captured image; and

a motion detection unit that detects a motion of a photographic subjectin the captured images.

The captured images may be images of a living body.

The motion detection unit may be caused to detect, as the motion,magnitudes of components of a motion vector on the basis of one of thecaptured images.

The light-source control unit may be caused to control the light sourcesuch that the light source outputs light beams containing the samewavelength component at times of the pulsed-light emissions.

The motion detection unit may be caused to further detect, as themotion, a direction of the motion vector on the basis of a plurality ofthe captured images.

The image processing device may further include

a motion correction unit that performs motion correction with respect tothe captured images on the basis of a result of the detection of themotion.

The image processing device may further include

an image generation unit that generates images of the photographicsubject from other ones of the captured images on the basis of a resultof the detection of the motion, the other ones of the captured imagesbeing at time points when the motion is not made.

The light-source control unit may be caused to control the light sourcesuch that exposure time periods in each of which the light beamcontaining the wavelength component is continuously output, and otherexposure time periods in each of which the pulsed-light emission isperformed the plurality of times are provided.

The motion detection unit may be caused to detect the motion on thebasis of ones of the captured images, the ones of the captured imagescorresponding to the other exposure time periods in each of which thepulsed-light emission is performed the plurality of times.

The image processing device may further include

a motion correction unit that performs motion correction with respect toother ones of the captured images on the basis of a result of thedetection of the motion, the other ones of the captured imagescorresponding to the exposure time periods in each of which the lightbeam containing the wavelength component is continuously output.

The light-source control unit may be caused to control the light sourcesuch that the exposure time periods in each of which the light beamcontaining the wavelength component is continuously output, and theother exposure time periods in each of which the pulsed-light emissionis performed the plurality of times are provided alternately to eachother.

The light-source control unit may be caused to control the light sourcesuch that the other exposure time periods in each of which thepulsed-light emission is performed the plurality of times are providedat unequal intervals.

The light-source control unit may be caused to control the light sourcesuch that the light source outputs light beams containing wavelengthcomponents different from each other respectively at times of theplurality of times of pulsed-light emissions.

The motion detection unit may be caused to detect, as the motion, amotion vector on the basis of images respectively containing thewavelength components, the images respectively containing the wavelengthcomponents being obtained from one of the captured images.

The light-source control unit may be caused

to control another light source different from the light source suchthat the other light source continuously outputs a light beam containinga predetermined wavelength component during an exposure time period ofeach input image of the photographic subject, the input images beingdifferent from the captured images, and

to control the light source such that the light source outputs lightbeams containing another wavelength component different from thepredetermined wavelength component by performing the pulsed-lightemission the plurality of times in a time period including at least apart of the exposure time period of each of the input images.

The image processing device may further include

a first imaging unit that takes the captured images,

a second imaging unit that takes the input images, and

a splitting element that

-   -   optically splits the light beams from the photographic subject,    -   inputs ones of the split light beams to the first imaging unit,        and    -   inputs other ones of the split light beams to the second imaging        unit.

The image processing device may further include

a motion correction unit that performs motion correction with respect tothe input images on the basis of a result of the detection of themotion.

The light-source control unit may be caused to control the light sourcesuch that the light source performs the pulsed-light emissions in aplurality of different periods while changing periods of thepulsed-light emissions.

The motion detection unit may be caused to detect, as the motion, avibration period of the photographic subject on the basis of degrees ofcontrasts of the captured images obtained respectively in the pluralityof different periods.

The light-source control unit may be caused to cause, after thedetection of the motion, the light source to perform the pulsed-lightemissions in a period in accordance with a result of the detection ofthe motion.

According to the first aspect of the present technology, there isprovided an image processing method or a program including the steps of:

controlling a light source such that the light source performs apulsed-light emission a plurality of times in an exposure time period ofeach captured image; and

detecting a motion of a photographic subject in the captured images.

According to the first aspect of the present technology,

the light source is controlled to perform the pulsed-light emission theplurality of times in the exposure time period of each of the capturedimages, and

the motion of the photographic subject in the captured images isdetected.

According to a second aspect of the present technology, there isprovided an endoscope system including:

a light source capable of performing a pulsed-light emission;

a light-source control unit that controls the light source such that thelight source performs the pulsed-light emission a plurality of times inan exposure time period of each captured image;

an imaging unit that takes the captured images; and

a motion detection unit that detects a motion of a photographic subjectin the captured images.

According to the second aspect of the present technology,

the light source is controlled to perform the pulsed-light emission theplurality of times in the exposure time period of each of the capturedimages,

the captured images are taken, and

the motion of the photographic subject in the captured images isdetected.

Advantageous Effects of Invention

According to the first aspect and the second aspect of the presenttechnology, the motion detection can be performed with higher accuracy.

Note that, the advantages disclosed herein are not necessarily limitedto those described hereinabove, and all the advantages describedhereinabove and hereinbelow can be obtained.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A diagram showing a configuration example of an endoscope system.

FIG. 2 A diagram showing a functional configuration example of theendoscope system.

FIG. 3 An explanatory chart showing pulsed-light emissions.

FIG. 4 An explanatory view illustrating motion detection based on aninput image.

FIG. 5 An explanatory view illustrating directions of a motion.

FIG. 6 An explanatory diagram showing generation of a motion informationitem by the motion detection.

FIG. 7 An explanatory flowchart showing an imaging procedure.

FIG. 8 An explanatory chart showing pulsed-light emissions in respectivecolors in one-frame time period.

FIG. 9 An explanatory view illustrating motion detection with use ofplain images.

FIG. 10 An explanatory flowchart showing another imaging procedure.

FIG. 11 An explanatory chart showing a normal light emission andpulsed-light emissions in one-frame time period.

FIG. 12 An explanatory flowchart showing still another imagingprocedure.

FIG. 13 Another explanatory chart showing the normal light emissions andthe pulsed-light emissions.

FIG. 14 An explanatory flowchart showing yet another imaging procedure.

FIG. 15 An explanatory chart showing the normal light emission andpulsed-light emissions in one-frame time period.

FIG. 16 A diagram showing a configuration example of an imaging unit.

FIG. 17 An explanatory flowchart showing yet another imaging procedure.

FIG. 18 A diagram showing another functional configuration example ofthe endoscope system.

FIG. 19 An explanatory view illustrating blurring of an input image ofvocal cords as a photographic subject.

FIG. 20 An explanatory flowchart showing yet another imaging procedure.

FIG. 21 An explanatory graph showing another example of an output image.

FIG. 22 A diagram showing a configuration example of an endoscopicsurgical system.

FIG. 23 A diagram showing a configuration example of a computer.

MODE(S) FOR CARRYING OUT THE INVENTION

Now, with reference to the drawings, embodiments to which the presenttechnology is applied are described.

First Embodiment

<Configuration Example of Endoscope System>

FIG. 1 is a diagram showing a configuration example of an embodiment ofan endoscope system to which the present technology is applied.

An endoscope system 11 shown in FIG. 1 includes a scope 21, a camerahead 22, a light source device 23, a camera control unit 24, anoperation input device 25, and a monitor 26. Note that, in thefollowing, the camera control unit 24 is referred to also as a CCU(Camera Control Unit) 24 as appropriate.

This endoscope system 11, which is used mainly in the medical field, isa system that functions as an endoscope device that causes the scope 21to be inserted into a participant, and takes images of arbitrary parts(surgical parts) in the participant as images of a surgical field. Notethat, in the following, description is made by way of an example inwhich a photographic subject is the participant such as a patient, but,as a matter of course, the photographic subject to be an imaging targetin the endoscope system 11 may be a living body other than humans.

The scope 21 is a lens barrel portion that includes an optical systemconstituted by lenses such as an objective lens, and that is insertedinto the participant. The scope 21 applies illumination light beamsinput from the light source device 23 to the photographic subject. Thescope 21 converges incident reflected-light beams from the photographicsubject, and guides the converged reflected-light beams to the camerahead 22.

Specifically, when the illumination light beams are applied from thescope 21 to the photographic subject, these illumination light beamsturn into the reflected light beams by being reflected by thephotographic subject. The scope 21 converges and inputs the reflectedlight beams into the camera head 22. Note that, the scope 21 may be aflexible lens barrel, or may be a rigid lens barrel.

The camera head 22, which is provided integrally with the scope 21, isconstituted, for example, by a camera, more specifically, by an imagingelement or the like. Under control by the CCU 24, the camera head 22captures the photographic subject by receiving and photoelectricallyconverting the incident reflected-light beams from the scope 21, andsupplies image data items of resultant input images to the CCU 24. Suchinput images are, for example, the images of the photographic subject,specifically, images of the surgical field at a time of performingsurgery or the like, more specifically, images of the living body suchas the patient.

The light source device 23, which is constituted, for example, by alaser light source or an LED (Light Emitting Diode) light source,outputs light beams in a specific wavelength band as the illuminationlight beams under the control by the CCU 24, and inputs theseillumination light beams to the scope 21. For example, from the lightsource device 23, white light beams or the like are output as theillumination light beams.

The CCU 24 control operations of an entirety of the endoscope system 11.The CCU 24 includes a light-source control device 31, a signalprocessing circuit 32, a detection unit 33, and a memory 34.

Under control by the signal processing circuit 32, the light-sourcecontrol device 31 controls the application of the illumination lightbeams by the light source device 23, that is, ON/OFF of the illuminationlight beams. Specifically, the light-source control device 31 controls,for example, irradiation timings and irradiation time periods of theillumination light beams, and light intensity of the illumination lightbeams.

Further, the light-source control device 31 supplies synchronizingsignals for synchronizing an illumination operation by the light sourcedevice 23, and the imaging operation by the camera head 22 with eachother to the camera head 22.

The signal processing circuit 32 supplies the input images supplied fromthe camera head 22 to the detection unit 33, and causes the detectionunit 33 to detect predetermined features. In addition, the signalprocessing circuit 32 generates appropriate signal-processing parameterson the basis of detection results supplied from the detection unit 33and of data items recorded in the memory 34. Further, on the basis ofthe signal-processing parameters, the signal processing circuit 32executes a predetermined signal process on the input images suppliedfrom the camera head 22, and supplies resultant output images to themonitor 26.

The detection unit 33 detects, from the input images supplied from thesignal processing circuit 32, the predetermined features such as amotion of the photographic subject in the input images, and supplies theresults of the detection to the signal processing circuit 32.

The memory 34 records, for example, various data items such as preparedconversion parameters, and the signal-processing results supplied fromthe signal processing circuit 32, and supplies, for example, data itemsrecorded therein to the signal processing circuit 32.

The operation input device 25, which includes buttons, switches, or atouchscreen superimposed on the monitor 26, is operated by a user whooperates the endoscope system 11. The operation input device 25 suppliessignals in response to the operations by the user to the signalprocessing circuit 32 and the light-source control device 31.

The monitor 26, which is constituted, for example, by a liquid-crystaldisplay panel, displays the output images supplied from the signalprocessing circuit 32.

<Functional Configuration Example of Endoscope System>

Next, a functional configuration example of the endoscope system 11shown in FIG. 1 is described. FIG. 2 is a diagram showing the functionalconfiguration example of the endoscope system 11. Note that, in FIG. 2,components corresponding to those in the case of FIG. 1 are denoted bythe same reference symbols, and description thereof is omitted asappropriate.

The endoscope system 11 shown in FIG. 2 includes a light-source controlunit 61, the light source device 23, an imaging unit 62, a motiondetection unit 63, an output-image generation unit 64, the monitor 26,and a recorder 65.

The light-source control unit 61, which corresponds, for example, to thelight-source control device 31 shown in FIG. 1, supplies thesynchronizing signals for synchronizing the application of theillumination light beams and the taking of the input images with eachother to the light source device 23 and the imaging unit 62.

The light source device 23 performs pulsed-light emissions atpredetermined timings in response to the synchronizing signals suppliedfrom the light-source control unit 61 so that the illumination lightbeams are applied to the photographic subject. When the illuminationlight beams are applied to the photographic subject, these illuminationlight beams turn into the reflected light beams by being reflected bythe photographic subject, and then are received by the imaging unit 62.

The imaging unit 62, which is constituted by the camera head 22 shown inFIG. 1, captures the photographic subject as captured images to be theinput images in response to the synchronizing signals supplied from thelight-source control unit 61, and supplies these input images to themotion detection unit 63 and the output-image generation unit 64. Inother words, the imaging unit 62 includes the imaging elementconstituting the camera, and obtains the input images by receiving andphotoelectrically converging the reflected light beams from thephotographic subject.

The motion detection unit 63 detects the motion of the photographicsubject in the input images on the basis of the input images suppliedfrom the imaging unit 62, and supplies, for example, resultant motionvectors as motion information items indicating detection results of themotion of the photographic subject to the output-image generation unit64. The detection unit 33 of FIG. 1 serves as the motion detection unit63, for example.

On the basis of the motion information items supplied from the motiondetection unit 63, the output-image generation unit 64 performs motioncorrection such as image stabilization with respect to the input imagessupplied from the imaging unit 62. Then, the output-image generationunit 64 supplies the resultant output images to the monitor 26 and therecorder 65. The signal processing circuit 32 of FIG. 1 serves as theoutput-image generation unit 64, for example. When the output-imagegeneration unit 64 performs the motion correction such as the imagestabilization in this way, the output-image generation unit 64 functionsas a motion correction unit.

The recorder 65, which is constituted, for example, by a nonvolatilerecording unit (not shown in FIG. 1) connected to the signal processingcircuit 32 of FIG. 1, records the output images supplied from theoutput-image generation unit 64. Further, the monitor 26 displays theoutput images supplied from the output-image generation unit 64.

<Processes in Endoscope System>

Next, an example of specific processes in the endoscope system 11 shownin FIG. 2 is described.

As the light source device 23, for example, a laser light source that iscapable of performing the pulsed-light emissions and emits RGBwhite-light beams is used.

In such a case, the light source device 23 includes an “R” light sourcethat outputs a narrow-wavelength light beam containing a wavelengthcomponent of R (red), a “G” light source that outputs anarrow-wavelength light beam containing a wavelength component of G(green), a “B” light source that outputs a narrow-wavelength light beamcontaining a wavelength component of B (blue), and a synthetic opticalsystem that synthesizes with each other and outputs the light beamsoutput from these light sources.

Thus, when the white light beam is output as the illumination light beamfrom the light source device 23, the light beams to be outputsimultaneously with each other from the “R” light source, the “G” lightsource, and the “B” light source, that is, the light beam containing the“R” component, the light beam containing the “G” component, and thelight beam containing the “B” component are synthesized with each otherby the synthetic optical system, and then output from the light sourcedevice 23. Note that, although the illumination light beam, whichcontains a predetermined wavelength component and is output from thelight source device 23, is the white light beam in the description ofthis example, the illumination light beam may be a light beam in anywavelength band.

As shown, for example, in FIG. 3, the light source device 23 emits thelight beam twice in a time period of one frame corresponding to theinput image. Note that, in FIG. 3, the abscissa axis represents time,and downward arrows in FIG. 3 each indicate a timing of the lightemission by the light source device 23, that is, a timing of theapplication of the illumination light beam to the photographic subject.

In this example, for example, a time period T1 represents the timeperiod corresponding to one frame of the input image, that is, anexposure time period for taking the input image corresponding to the oneframe. In the time period T1 corresponding to the one frame, the lightsource device 23 performs the pulsed-light emission at each time point,that is, a time point t1 and a time point t2.

In particular, in this example, at the times of the pulsed-lightemissions, light beams containing the same wavelength component, thatis, in this example, the white light beams are output from the lightsource device 23. Further, light-emission control is performed such thatthe light source device 23 performs the pulsed-light emissions atcertain time intervals, that is, in certain periods. The pulsed-lightemission by the light source device 23 is performed twice at an equalinterval in each of the time periods corresponding to the frames.

For example, when the input images are taken as those of a 60P movingimage, the one-frame time period of the input images is 1/60 s. Thus,the light source device 23 emits the light beams at timings of every1/120 s.

In response to the synchronizing signals supplied from the light-sourcecontrol unit 61, the light source device 23 performs the pulsed-lightemissions of the white light beams as the illumination light beams atthe intervals of 1/120 s. When the illumination light beams are appliedto the photographic subject, the reflected light beams from thephotographic subject enter the imaging unit 62. Thus, the imaging unit62 performs an imaging operation of receiving and photoelectricallyconverting the illumination light beams at the timings indicated by thesynchronizing signals from the light-source control unit 61.

In this case, for example, the time period T1 is set as the exposuretime period, and the imaging unit 62 takes the input image of the framecorresponding to the time period T1. The image data item of theresultant input image is supplied to the motion detection unit 63 andthe output-image generation unit 64.

The motion detection unit 63 performs motion detection with respect tothe input image supplied from the imaging unit 62, and supplies theresultant motion-information items such as the motion vector to theoutput-image generation unit 64.

Specifically, in the case where the illumination light beam is appliedtwice in the one-frame time period of the input image as shown in FIG.3, when the input images depict a moving photographic subject, an imageof the photographic subject is a double image as illustrated, forexample, in FIG. 4.

Note that, when the input images depict the moving photographic subject,the photographic subject itself may be moving, that is, the photographicsubject may be an active photographic subject, or the imaging unit 62may be moving with respect to the photographic subject due to camerashake or the like. In other words, the motion of the photographicsubject in the input images obtained as a result of the motion detectionwith respect to the input images is a motion of the photographic subjectitself, a motion of the photographic subject with respect to the imagingunit 62, or both the motions.

In the example of FIG. 4, a photographic-subject image MH11-1 and aphotographic-subject image MH11-2 being images of the same photographicsubject are contained in the input images. Note that, in the following,unless it is necessary to make a specific distinction between thephotographic-subject image MH11-1 and the photographic-subject imageMH11-2, these images are simply referred to also as photographic-subjectimages MH11.

For example, when the input images of FIG. 4 are taken in the timeperiod T1 shown in FIG. 3, one of the two photographic-subject imagesMH11 is an image corresponding to the reflected light beam that entersthe imaging unit 62 at a timing of the time point t1 shown in FIG. 3,and another one of the photographic-subject images MH11 is an imagecorresponding to the reflected light beam that enter the imaging unit 62at a timing of the time point t2.

When the light source device 23 performs the pulsed-light emission twicein the exposure time period in this way, the two images of the samephotographic subject are contained in the input images. However, whichof these two photographic-subject images corresponds to which of thetime points cannot be distinguished.

The motion detection unit 63 sets one desired region in an input imagecorresponding to a processing-target frame as an attention region AR11,and sets a predetermined rectangular region around the attention regionAR11 as a search range SR11. In other words, the search range SR11 beinga search region is set as a rectangular region corresponding to upper,lower, right, and left detection-target shift ranges with respect to theattention region AR11 in FIG. 4.

The motion detection unit 63 sets regions each having the same size asthat of the attention region AR11 at positions in the search range SR11as comparison regions, and performs search while shifting the comparisonregions.

In this example, the search is performed in an order of raster scan froman upper-left comparison region CR11-1 in the search range SR11 in FIG.4 to a lower-right comparison region CR11-2 in the search range SR11 inFIG. 4. Note that, in the following, unless it is necessary to makespecific distinctions between the comparison regions such as thecomparison region CR11-1 and the comparison region CR11-2 in the searchrange SR11, these regions are simply referred to also as comparisonregions CR11.

The motion detection unit 63 calculates an autocorrelation coefficientbetween the attention region AR11 and each of the comparison regionsCR11.

Specifically, the motion detection unit 63 calculates an autocorrelationcoefficient R from the following equation (1) where a pixel value of ani-th pixel in the attention region AR11 is Xi, and a pixel value of ani-th pixel in the comparison region CR11 is Yi.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{11mu} 1} \right\rbrack\mspace{619mu}} & \; \\{R = \frac{\sum\limits_{i}{\left( {{Xi} - {Xa}} \right)\left( {{Yi} - {Ya}} \right)}}{\sqrt{\sum\limits_{i}\left( {{Xi} - {Xa}} \right)^{2}}\sqrt{\sum\limits_{i}\left( {{Yi} - {Ya}} \right)^{2}}}} & (1)\end{matrix}$

Note that, in the equation (1), Xa is an average value of the pixelvalues of all pixels in the attention region AR11, and Ya is an averagevalue of the pixel values of all pixels in the comparison region CR11.

When such an autocorrelation coefficient R is calculated with respect toall the comparison regions CR11 in the search range SR11, theautocorrelation coefficient R reaches its maximum value, that is, theautocorrelation coefficient R=1 is obtained when the comparison regionCR11 is identical to the attention region AR11.

Further, when the comparison region CR11 corresponds to a second-largestlocal maximum value of the autocorrelation coefficient R, the comparisonregion CR11 at this time is a region depicting the same photographicsubject as that in the attention region AR11. In other words, thecomparison region CR11 at this time is a region shifted from theattention region AR11 by an amount of the motion vector indicating amotion of the photographic subject in the attention region AR11.

In this example, the second-largest local maximum value of theautocorrelation coefficient R is obtained by a combination of theattention region AR11 and the comparison region CR11-3. Thus, an arrow(vector) connecting a center of the attention region AR11 and a centerof the comparison region CR11-3 to each other, in other words,displacement between the attention region AR11 and the comparison regionCR11-3 corresponds to the motion vector of the attention region AR11.

In this way, the motion detection unit 63 calculates the motion of thephotographic subject with respect to the attention region in the inputimage corresponding to the processing-target frame.

Note that, when the motion is calculated from the images of the samephotographic subject contained in the input images, that is, from thedouble image of the photographic subject, magnitudes of components of atwo-dimensional motion vector can be calculated, but directions of themotion vector cannot be calculated.

For example, as illustrated in FIG. 5, with respect to the attentionregion AR11, the comparison region CR11-3 is calculated as the regioncorresponding to the second-largest local maximum value of theautocorrelation coefficient R. Note that, in FIG. 5, componentscorresponding to those in the case of FIG. 4 are denoted by the samereference symbols, and description thereof is omitted as appropriate.

In this case, the vector connecting the attention region AR11 and thecomparison region CR11-3 to each other corresponds to the motion vector.However, whether the photographic subject has moved from a position ofthe attention region AR11 to a position of the comparison region CR11-3as indicated by an arrow MV11, or the photographic subject has movedfrom the position of the comparison region CR11-3 to the position of theattention region AR11 as indicated by an arrow MV12 cannot bedistinguished from the input image corresponding to one frame.

As a countermeasure, when the direction of the motion is needed as themotion information item, it is only necessary to calculate the directionof the motion with use of input images corresponding to a plurality offrames, specifically, for example, the input image corresponding to theprocessing-target frame and an input image corresponding to a frameimmediately preceding the processing-target frame. In such a case, thedirection of the motion can be calculated by utilizing the same methodas those at times of detecting general inter-frame motion vectors.

For example, as shown in FIG. 6, it is assumed that there are a frame2being a frame corresponding to a latest time point, and a frame1 being aframe immediately preceding the frame2.

The motion detection unit 63 performs the motion detection in each ofthe frame1 and the frame2 so as to calculate a magnitude of a motionvector of the photographic subject in each of input images correspondingto these frames, that is, magnitudes of components in directions such asan x-direction and a y-direction of the motion vector.

With respect to these magnitudes, the motion detection unit 63 performsmotion detection such as inter-frame block matching with use of theinput images corresponding respectively to the frame1 and the frame2,and determines a direction (orientation) of a resultant motion vector asa direction of a motion of the photographic subject in the latestframe2.

Then, the motion detection unit 63 determines the motion vector that iscalculated from the magnitudes of the motion vectors, which are obtainedas described above, that is, from the magnitude of the component in eachof the directions, and from the direction (orientation) of the motionvector as the motion information item indicating the detected motion.

Normally, when the motion detection is performed with respect to theinput images, and the motion correction such as the image stabilizationis performed with use of the resultant motion-information item, a motionbetween the frame1 and the frame2 is detected, and a result of thedetection is applied to a frame subsequent to the frame2 (hereinafter,referred to as frame3). However, in this case, an information item ofthe frame1 that is temporally away from the frame3 is used, and hencethe motion detection cannot be performed with high accuracy.

In contrast, in the endoscope system 11, the motion is detected onlyfrom the frame2 immediately preceding the frame3 to which the result ofthe motion detection is applied is detected. Thus, the motion detectionis performed with use of an information item newer than that at the timeof using the immediately-preceding two frames. For example, at a time ofestimating a motion in the frame3, accuracy in motion detection becomeshigher by using an information item closer to the frame3. This provesthat the motion detection can be performed with higher accuracy at thetime of detecting the motion only from the frame2 than at the time ofusing the immediately-preceding two frames.

Note that, at the time of obtaining the motion vector as the motioninformation item, also in the endoscope system 11, not only the frame2but also the preceding frame1 is used for detecting the direction of themotion vector. However, the orientation of the motion vector is scarcelyabruptly reversed. Thus, even when the frame1 and the frame2 are usedfor calculating the direction of the motion vector, accuracy indetection of the resultant motion vector as the motion information itemdoes not decrease.

In the endoscope system 11, the direction (orientation) of the motionvector that scarcely changes is detected with use of theimmediately-preceding two frames. Magnitudes of components of the motionvector, which have a significant influence on the detection accuracy,that is, minor motion changes are detected with use of only the latestframe. With this, the motion detection can be performed with higheraccuracy.

Note that, although the motion vector with respect to the one attentionregion in the input image is calculated as the motion information itemin the above-described example, the motion vector may be calculated asthe motion information item in each of a plurality of regions in anentirety of the input image. Alternatively, one motion vector may becalculated as the motion information item in the entirety of the inputimage.

Further, although the method of calculating the autocorrelationcoefficient is described as an example of methods of calculating themotion vector, the motion vector may be calculated by other matchingmethods such as a gradient method.

The motion detection unit 63 supplies, as the motion information item,the motion vector calculated as described above to the output-imagegeneration unit 64.

The output-image generation unit 64 generates the output image byperforming, for example, the motion correction for correcting a motionamount with respect to the input image with use of the motioninformation item supplied from the motion detection unit 63, the inputimage being supplied from the imaging unit 62, the motion amount beingcalculated from the motion information item. As the motion correction,the image stabilization is performed, for example.

Specifically, the output-image generation unit 64 performs the imagestabilization with respect, for example, to the input image on the basisof the motion vector in the processing-target frame, which is estimatedfrom the motion vector as the motion information item obtained from theframe immediately preceding the processing-target frame. In this case,for example, the input image corresponding to the processing-targetframe is shifted by an amount of the magnitude of the motion vector intoa direction opposite to the orientation of this motion vector, and thenis generated as the output image.

Further, alternatively, a correction of the double image contained inthe input image may be performed as the motion correction. As describedabove, in the operation of taking the input image, the pulsed-lightemission is performed twice, that is, multiple exposure is performed.Thus, when the photographic subject has moved, two images of thephotographic subject are contained in the input image. In this case, theoutput-image generation unit 64 may execute, on the input image, animage process of removing one of the two images of the same photographicsubject with use of the motion information item, the two images beingcontained in the input images, and then output another one of the twoimages as the output image.

The output-image generation unit 64 supplies the output image generatedin this way to the recorder 65, and causes the recorder 65 to recordthis output image. The output-image generation unit 64 also suppliesthis output image to the monitor 26, and causes the monitor 26 todisplay this output image.

As described above, in the endoscope system 11, the magnitudes of thecomponents of the motion vector as the motion information item can becalculated from the input image corresponding to the latest one frame,that is, from one input image. With this, the motion detection of thephotographic subject in the input image can be performed with higheraccuracy.

Further, in the endoscope system 11, the calculated motion informationitem can be immediately utilized for the motion correction process onthe subsequent frame. Thus, the output image can be obtained whilereducing a delay to a minimum.

In addition, as another example, the output-image generation unit 64 maydetect a time point (frame) when the photographic subject does not makethe motion from the motion information item of the attention region,which is obtained by the motion detection unit 63, and then generate anoutput image being an image for observing the photographic subject froman input image taken at the time point when the photographic subjectdoes not make the motion. Alternatively, when a state in which thephotographic subject does not make the motion is detected, the signalprocessing circuit 32 may control the imaging unit 62 such that theimaging unit 62 immediately takes a new input image, for example, by anormal light emission under the state in which the photographic subjecthas not moved, and that the output image for the observation isgenerated from the obtained input image.

In these cases, the image of the photographic subject is not doubled inthe input image. Thus, the motion correction is unnecessary, and anunblurred still image can be obtained as the output image.

The time point when the motion is not made, for example, a time pointwhen the motion vector as the motion information item is zero orsubstantially zero. Such a time point is, for example, a time point whenthere is no comparison region corresponding to the second-largest localmaximum value of the autocorrelation coefficient, or a time point whenthe second-largest local maximum value of the autocorrelationcoefficient is equal to or less than a predetermined threshold.

Further, when the output image under the state in which the photographicsubject does not make the motion is to be obtained, it is only necessaryto grasp only presence/absence of the motion, and hence the direction ofthe motion is unnecessary. Therefore, in such a case, only themagnitudes of the components of the motion vector may be detected as themotion information item.

The method of generating the output image by detecting the time point atwhich the motion is not made is particularly advantageous, for example,at a time of acquiring the diagnostic unblurred still image.

The endoscope system 11 as described above is particularly advantages,for example, at the time of executing the image stabilization process,or at the time of acquiring a motionless still image.

When the motion of the photographic subject is dynamic, accuracy innormal motion detection of the dynamic motion decreases due to motionblurring. However, in the endoscope system 11, the pulsed-lightemissions are performed at the time of the exposure. Thus, blurring ofthe input image to be acquired can be reduced, and the motion detectioncan be performed with high accuracy even with respect to rapid motions.Further, when the pulsed-light emissions are performed, a substantialexposure time period is shortened. Thus, even motions that rapidlychange, and vibrating motions in short periods also can be detected withhigh accuracy.

<Imaging Procedure>

Next, with reference to a flowchart of FIG. 7, a procedure in which theendoscope system 11 generates the output images is described. In otherwords, in the following, with reference to the flowchart of FIG. 7, animaging procedure by the endoscope system 11 is described.

In Step S11, the light source device 23 performs the pulsed-lightemissions under control by the light-source control unit 61.Specifically, the light source device 23 performs the pulsed-lightemission twice per frame time period at the timings indicated by thesynchronizing signals from the light-source control unit 61.

In Step S12, the imaging unit 62 takes the input images in response tothe synchronizing signals supplied from the light-source control unit61.

Specifically, the imaging unit 62 obtains the input images by receivingand photoelectrically converting the incident reflected-light beams. Inthis example, the pulsed-light emission is performed twice per frametime period, that is, per exposure time period. Thus, the input imagesare each a multiple-exposure image containing the images of thephotographic subject at the times of the respective pulsed-lightemissions. The imaging unit 62 supplies the obtained input images to themotion detection unit 63 and the output-image generation unit 64.

In Step S13, the motion detection unit 63 performs the intra-framemotion detection on the basis of each of two of the input imagessupplied from the imaging unit 62 and each corresponding to one frame.

Specifically, the motion detection unit 63 detects the comparison regioncorresponding to the second-largest local maximum value by performingthe calculation using the above-described equation (1) with respect tothe attention region in one of the two of the input images. In this way,the motion detection unit 63 calculates the magnitudes of the componentsof the motion vector in the processing-target frame.

In Step S14, the motion detection unit 63 performs the inter-framemotion detection on the basis of the input image corresponding to theprocessing-target frame and the input image corresponding to the frameimmediately preceding the processing-target frame.

Specifically, the motion detection unit 63 detects the direction(orientation) of the motion vector by performing, for example, the blockmatching on the basis of the input images corresponding to theimmediately-preceding two frames. Note that, the present technology isnot limited to the example described hereinabove in which the directionof the motion vector as the motion of the photographic subject in theinput images is detected with use of the input images corresponding tothe immediately-preceding two frames. However, the direction of themotion vector may be detected with use of a plurality of differentframes, that is, input images at a plurality of different time points.

By the processes of these Step S13 and Step S14, the motion vector inthe processing-target frame is obtained. The motion detection unit 63supplies, as the motion information item, the motion vector obtained inthis way to the output-image generation unit 64.

Note that, more specifically, the processes of Step S13 and Step S14 areexecuted simultaneously, that is, concurrently with each other.

In Step S15, the output-image generation unit 64 performs, on the basisof the motion information item supplied from the motion detection unit63, the motion correction with respect to the input image supplied fromthe imaging unit 62, and then generates the output image.

For example, in Step S15, the image stabilization process on the inputimage corresponding to the processing-target frame is executed as themotion correction on the basis of the motion information item obtainedfrom the frame immediately preceding the processing-target frame, andthen the output image is generated. In this case, the motion informationitem calculated in Step S13 and Step S14 is used in a subsequent frame.Thus, the process of Step S15 may be executed simultaneously with theprocesses of these Step S13 and Step S14. Alternatively, the process ofStep S15 may be executed before the execution of the processes of StepS13 and Step S14. Note that, when real-time characteristics are notrequired, the motion correction with respect to the input imagecorresponding to the processing-target frame may be performed with useof a motion information item obtained from this processing-target frame.

In Step S16, the output-image generation unit 64 outputs the outputimage obtained by the process of Step S15 to the recorder 65 or themonitor 26. Then, the imaging procedure is ended. With this, the outputimage is recorded in the recorder 65, or the output image is displayedon the monitor 26.

As described above, by performing the pulsed-light emissions a pluralityof times in one-frame time period, the endoscope system 11 detects amotion, specifically, the magnitudes of the components of the motionvector from the input image corresponding to one frame. With this, themotion detection can be performed with higher accuracy and with a smalldelay.

Second Embodiment

<Motion Detection>

Further, in the example described in the first embodiment, the laserlight source that emits the RGB white-light beams is used as the lightsource device 23. However, the light source device 23 may be, forexample, a laser light source capable of sequentially performingpulsed-light emissions of laser light beams respectively containingcolor components of R, G, and B.

In such a case, as shown in FIG. 8, the light source device 23sequentially performs the pulsed-light emissions of the light beamsrespectively containing the color components of R (Red), G (Green), andB (Blue), for example, once each in a time period corresponding to oneframe. In other words, the light-source control unit 61 performslight-emission control such that, respectively at times of the pluralityof times of pulsed-light emissions in the time period corresponding toone frame, the light beams containing the color components differentfrom each other, that is, light beams containing wavelength componentsdifferent from each other are output from the light source device 23.Note that, in FIG. 8, the abscissa axis represents time.

In this embodiment, the light source device 23, which includes the “R”light source, the “G” light source, and the “B” light source, causesthese light sources to sequentially and independently perform thepulsed-light emissions. With this, the light beam containing the “R”component, the light beam containing the “G” component, and the lightbeam containing the “B” component can be sequentially emitted as theillumination light beams.

In FIG. 8, a section indicated by an arrow Q11 corresponds to thesynchronizing signals for the frames corresponding to the input images.Specifically, in FIG. 8, in the section indicated by the arrow Q11,concave-upward parts correspond to boundary positions between theframes.

Further, sections indicated respectively by an arrow Q12 to an arrow Q14correspond respectively to exposure timings of the light beamscontaining the “R” components, the light beams containing the “G”components, and the light beams containing the “B” components. Forexample, in FIG. 8, in the section indicated by the arrow Q12,concave-downward time periods each correspond to one-frame exposure timeperiod of the light beam containing the “R” component. In particular, inthis example, irrespective of the frames, the light beams containing the“R” components, the light beams containing the “G” components, and thelight beams containing the “B” components are exposed at theirrespective common timings.

In addition, a section indicated by an arrow Q15 corresponds to lightemission timings of the pulsed-light emissions by the light sourcedevice 23. Specifically, solid-line parts each correspond to a lightemission timing of the light beam containing the “R” component from the“R” right source, dotted-line parts each correspond to a light emissiontiming of the light beam containing the “G” component from the “G” rightsource, and dash-dotted-line parts each correspond to a light emissiontiming of the light beam containing the “B” component from the “B” rightsource.

In this example, a time period T21 corresponds, for example, to the timeperiod of one frame corresponding to the input image, and thepulsed-light emission is performed three times in this time period T21.Specifically, first, at a time point t21, the light beam of the “R”color, that is, the illumination light beam containing the “R” componentis output by the pulsed-light emission. Then, at a time point t22, theillumination light beam containing the “G” component is output by thepulsed-light emission. Next, at a time point t23, the illumination lightbeam containing the “B” component is output by the pulsed-lightemission.

For example, when the input images are taken as those of the 60P movingimage, the one-frame time period of the input images is 1/60 s. Thus,the light source device 23 emits the light beams in an order of R, G,and B at timings of every 1/180 s. Specifically, focusing on the colorcomponents of the light beams, the light beams of the same color areemitted at the timings of every 1/60 s.

Further, in the imaging unit 62, exposure is performed at appropriatetimings in accordance with the light emission timings of the lightsource device 23. When the imaging unit 62 is, for example, aBayer-array single-plate sensor, on an imaging surface of the imagingunit 62, color filters that transmit therethrough only the light beamsrespectively containing the color components of R, G, and B are providedrespectively to the pixels. Specifically, on the imaging surface of theimaging unit 62, “R” pixels that receive only the light beams containingthe “R” components, “G” pixels that receive only the light beamscontaining the “G” components, and “B” pixels that receive only thelight beams containing the “B” components are provided.

In the example of FIG. 8, all the color components are exposed at theirrespective common timings, and an image to be obtained by the exposurecorresponding to one frame is an image of what is called a RAW data itemincluding the “R” pixels, the “G” pixels, and the “B” pixels(hereinafter, referred to as RAW image).

Thus, when the “R” pixels are extracted from the RAW image, and aninterpolation process and the like are executed thereon as appropriate,an “R” image formed only of the “R” pixels, that is, having values ofonly the “R” components is obtained. Similarly, when the “G” pixels areextracted from the RAW image, and the interpolation process and the likeare executed thereon as appropriate, a “G” image formed only of the “G”pixels is obtained. When the “B” pixels are extracted from the RAWimage, and the interpolation process and the like are executed thereonas appropriate, a “B” image formed only of the “B” pixels is obtained.

In other words, by taking the RAW image, the “R” image is obtained from“R” signals being pixel signals output from the “R” pixels, the “G”image is obtained from “G” signals being pixel signals output from the“G” pixels, and the “B” image is obtained from “B” signals being pixelsignals output from the “B” pixels. In still other words, by executing ademosaic process on one RAW image, plain images respectively containingthe color components of R, G, and B, that is, the “R” image, the “G”image, and the “B” image are obtained from the RAW image.

When the imaging unit 62 includes the pixels of the color components,although the light emission timings of the light beams respectivelycontaining the color components are different from each other, theexposure time periods of the color components can be set equal to eachother, specifically, to the time period corresponding to one frame. Inother words, by performing the pulsed-light emissions at timingsdifferent from each other correspondingly respectively to the colorcomponents in the one-frame time period, the plain images thatrespectively containing the color components and images at the timepoints different from each other can be obtained from the taken RAWimage corresponding to one frame.

Note that, although, in FIG. 8, for the sake of simplicity ofdescription, description is made by way of an example in which globalshutter imaging, that is, all the pixels in the imaging unit 62 areexposed at once, the “R” image, the “G” image, and the “B” image can beobtained in the same way also when rolling shutter imaging, that is,exposures and readout are performed in each pixel row.

Substantial imaging time points of the plain images respectivelycontaining the three color components, which are obtained from the RAWimage corresponding to one frame in this way, are shifted from eachother by a time period of ⅓ frames. Thus, the motion detection unit 63performs the intra-frame motion detection with use of these “R” image,“G” image, and “B” image in the same frame so as to detect the motion ofthe photographic subject in the attention region.

In this case, as illustrated in FIG. 9, for example, three plain images,that is, an “R” image P11, a “G” image P12, and a “B” image P13 areobtained by imaging in the time period corresponding to one frame.

In this example, a photographic-subject image MH21-1 is observed in the“R” image P11, a photographic-subject image MH21-2 is observed in the“G” image P12, and a photographic-subject image MH21-3 is observed inthe “B” image P13. These photographic-subject image MH21-1 tophotographic-subject image MH21-3 are images of the same photographicsubject at the different time points. Note that, in the following,unless it is necessary to make specific distinctions between thephotographic-subject image MH21-1 to the photographic-subject imageMH21-3, these images are simply referred to also as photographic-subjectimages MH21.

The “R” image P11 to the “B” image P13 are the images at the time pointsdifferent from each other. Thus, when these images are superimposed oneach other, as illustrated, for example, in a lower part of FIG. 9, itis apparent that the photographic-subject images MH21 observed in theimages are slightly shifted from each other.

The motion detection unit 63 calculates a motion vector of an attentionregion AR21 with use of the “R” image P11 to the “B” image P13 and by,for example, a general motion-detection technique.

Specifically, the motion detection unit 63 detects, for example, acomparison region from each of the “G” image P12 and the “B” image P13,the comparison region corresponding to a maximum of an autocorrelationcoefficient with respect to the desired attention region AR21 in the “R”image P11.

Then, the motion detection unit 63 calculates the motion vector of thephotographic subject in the attention region AR21 from a positionalrelationship between the attention region AR21 in the “R” image P11, thecomparison region in the “G” image P12, which corresponds to the maximumof the autocorrelation coefficient, and the comparison region in the “B”image P13, which corresponds to the maximum of the autocorrelationcoefficient.

Note that, in this example, an order relationship between the timepoints of the photographic-subject images contained respectively in the“R” image P11 to the “B” image P13 has been already grasped. Thus, notonly magnitudes of components of the motion vector, but also a directionof the motion can be obtained. Thus, the motion vector can be calculatedonly from an information item of one frame.

<Another Imaging Procedure>

Next, a procedure that is executed in the endoscope system 11 at thetime when the pulsed-light emissions of the light beams respectivelycontaining the color components of R, G, and B are sequentiallyperformed is described. In other words, in the following, with referenceto the flowchart of FIG. 10, another imaging procedure by the endoscopesystem 11 is described.

In Step S41, the light source device 23 performs the pulsed-lightemissions corresponding respectively to the color components of R, G,and B under the control by the light-source control unit 61.

Specifically, the light source device 23 performs the pulsed-lightemission three times in total per frame time period, specifically, oncefor each of the colors of R, G, and B in this order at the timingsindicated by the synchronizing signals from the light-source controlunit 61.

In Step S42, the imaging unit 62 takes the input images in response tothe synchronizing signals supplied from the light-source control unit61. Specifically, the imaging unit 62 takes the RAW images as the inputimages by receiving and photoelectrically converting the incidentreflected-light beams. Then, the imaging unit 62 obtains, on the basisof each of the RAW images, the “R” image formed only of the “R” pixels,the “G” image formed only of the “G” pixels, and the “B” image formedonly of the “B” pixels as the plain images. Next, the imaging unit 62supplies the plain images respectively containing these color componentsas final input images to the motion detection unit 63 and theoutput-image generation unit 64. Note that, the RAW images may be outputas the input images such that, for example, in the output-imagegeneration unit 64 in a subsequent stage, the plain images respectivelycontaining the color components are generated from the RAW images.

In Step S43, the motion detection unit 63 calculates the motion vectorby performing the intra-frame motion detection on the basis of each ofthe input images supplied from the imaging unit 62 and eachcorresponding to one frame, specifically, on the basis of the “R” image,the “G” image, and the “B” image corresponding to one frame. Then, themotion detection unit 63 supplies the motion vector as the motioninformation item to the output-image generation unit 64.

After the motion information item is obtained, processes of Step S44 andStep S45 are executed, and then the imaging procedure is ended. Theseprocesses are the same as the processes of Step S15 and Step S16 in FIG.7, and hence description thereof is omitted.

Note that, in Step S44, for example, the image stabilization process isexecuted as the motion correction on each of the plain imagesrespectively containing the color components as the input images. Bysynthesizing the plain images respectively containing the colorcomponents after the image stabilization, a color output image includingthe pixels of the color components of R, G, and B is generated.

As described above, the endoscope system 11 performs the pulsed-lightemissions at the timings different from each other correspondinglyrespectively to the plurality of colors in one-frame time period,detects the motion from the input image corresponding to one frame, andgenerates the output image with use of the result of the detection. Withthis, the motion detection can be performed with higher accuracy andwith a small delay.

Third Embodiment

<Motion Detection>

Incidentally, in the above-described example, the motion of thephotographic subject is detected from the input images for generatingthe observational output images. However, images to be used for themotion detection (hereinafter, referred to also as detection images) maybe generated independently of the input images for generating the outputimages. More specifically, the input images and the detection images maybe generated as images at different time points of the same movingimage.

In such a case, as shown in FIG. 11, it is only necessary, for example,to sequentially separate a time period for the motion detection, thatis, a field for the detection image, and a time period for the normallight, that is, a field for the input image from each other.

Note that, in FIG. 11, components corresponding to those in the case ofFIG. 2 are denoted by the same reference symbols, and descriptionthereof is omitted as appropriate. Further, in FIG. 11, the abscissaaxis represents time, and downward arrows in FIG. 11 indicate timings ofthe light emissions by the light source device 23.

In the example shown in FIG. 11, a time period of one framecorresponding to the input image (output image) is divided into the twosequential fields. Specifically, a time period T31 is, for example, thetime period of one frame corresponding to the input image, and this timeperiod T31 is divided into two time periods, specifically, a time periodT41 and a time period T42.

Here, the time period T41 is set as a time period corresponding to onefield for obtaining the input image, that is, a time period for exposingthe input image. During the time period T41, the light source device 23performs a normal light emission of the white light beam. In otherwords, during the time period T41, the light source device 23 continuesto apply the white light beam to the photographic subject.

In this example, as in the first embodiment, the light source device 23is the laser light source that emits the white light beams. However, alight source for emitting the illumination light beam at the time of thenormal light emission, and a light source for performing thepulsed-light emissions of the illumination light beams for the motiondetection may be provided independently of each other.

For example, when the input images are taken by causing the light sourcedevice 23 to perform the pulsed-light emissions, light intensity duringthe exposure time period is lower than that at a time when the whitelight beam is continuously emitted. Thus, brightness may be insufficientfor observing the photographic subject in the input image.

As a countermeasure, in the example shown in FIG. 11, the time periodT41 that is half of the time period T31 of one frame corresponding tothe input image is set as the time period of one field. During this timeperiod of one field, the white light beam is continuously output fromthe light source device 23. In addition, the time period T41 is set asthe time period for exposing the input image.

Further, the time period T42 subsequent to the time period T41 is set asa time period of another one field in which, as in the first embodiment,the light source device 23 is caused to perform the pulsed-lightemission twice. Specifically, in the time period T42, first, at a timepoint t31, the light source device 23 is caused to perform thepulsed-light emission such that the white light beam is applied to thephotographic subject. Then, at a time point t32, the light source device23 is caused to perform the pulsed-light emission again such that thewhite light beam is applied to the photographic subject.

The time period T42 corresponding to the other one field in which thepulsed-light emission is performed twice in this way is set not as theexposure time period of the input image, but as an exposure time periodof the detection image for detecting the motion of the photographicsubject (attention region).

Thus, for example, when a frame rate of the output images is 60 fps, alength of the time period T31 corresponding to one frame is set to 1/60s, and the time period T41 and the time period T42 are each set to 1/120s.

From another perspective, in this example, a first exposure time periodin which a light beam containing a predetermined wavelength component,that is, the white light beam is continuously output, and a secondexposure time period in which the pulsed-light emission of the whitelight beam is performed a plurality of times are provided in the timeperiod corresponding to one frame. Thus, it can be said that two imagesare taken in this time period corresponding to one frame. In addition,the image obtained in the first exposure time period is used as theinput image for generating the output image, and the image obtained inthe second exposure time period is used as the detection image fordetecting the motion.

Note that, although the one-frame time period is divided into the twofields in the example described in this embodiment, the time period forobtaining the input image, and the time period for obtaining thedetection image may each be set as one-frame time period, and these timeperiods may be provided alternately to each other.

In this case, among the plurality of frames constituting the takenmoving image, frames each corresponding the first exposure time periodin which the white light beam is continuously output are used as theinput images, and frames each corresponding to the second exposure timeperiod in which the pulsed-light emission of the white light beam isperformed a plurality of times are used as the detection images.

In the example shown in FIG. 11, light-emission control and imagingcontrol are performed such that time periods each corresponding to oneframe, which is constituted by the time period corresponding to onefield for obtaining the input image and by the subsequent time periodcorresponding to the other one field for obtaining the detection image,are arranged in series in a time direction. In other words, thelight-emission control and the imaging control are performed such thatthe fields for obtaining the input images and the fields for obtainingthe detection images are arranged alternately to each other.

In this case, in the fields in each of which the normal light emissionis performed, the images taken by the imaging unit 62 are supplied asthe input images to the output-image generation unit 64. In contrast, inthe fields in each of which the pulsed-light emissions are performed,the images taken by the imaging unit 62 are supplied as the detectionimages to the motion detection unit 63.

Then, in the motion detection unit 63, the motion vector is calculatedas in the case of the first embodiment from the supplied detectionimages, and the obtained motion vector is supplied as the motioninformation item to the output-image generation unit 64.

Further, in the output-image generation unit 64, on the basis of themotion information item supplied from the motion detection unit 63, themotion correction such as the image stabilization process are performedas appropriate with respect to the input images supplied from theimaging unit 62. In this way, the output images are generated.

<Still Another Imaging Procedure>

Next, a procedure that is executed in the endoscope system 11 at thetime when the one-frame time period is divided into the field forobtaining the input image and the field for obtaining the detectionimage is described. In other words, in the following, with reference tothe flowchart of FIG. 12, still another imaging procedure by theendoscope system 11 is described.

In Step S71, the light source device 23 performs the normal lightemissions under the control by the light-source control unit 61.Specifically, the light source device 23 continuously emits and appliesthe white light beam to the photographic subject during the time periodcorresponding to one frame at the timing indicated by the synchronizingsignal from the light-source control unit 61.

In Step S72, the imaging unit 62 takes the input images in response tothe synchronizing signals supplied from the light-source control unit61, and supplies the obtained input images to the output-imagegeneration unit 64.

In Step S73, the light source device 23 performs the pulsed-lightemissions under the control by the light-source control unit 61.Specifically, in Step S73, the process similar to that of Step S11 inFIG. 7 is executed, more specifically, the pulsed-light emission isperformed twice in the one-field time period.

In Step S74, the imaging unit 62 takes the detection images in responseto the synchronizing signals supplied from the light-source control unit61.

Specifically, the imaging unit 62 obtains each of the detection imagesby receiving and photoelectrically converting the incidentreflected-light beams in the exposure time period being an entirety or apart of the one-field time period in which the pulsed-light emissionsare performed. In this example, the pulsed-light emission is performedtwice per field time period. Thus, the detection images are each themultiple-exposure image containing the images of the photographicsubject at the times of the respective pulsed-light emissions. Theimaging unit 62 supplies the obtained detection images to the motiondetection unit 63.

After corresponding one of the input images and corresponding one of thedetection images are obtained in the time period corresponding to oneframe in this way, processes of Step S75 to Step S78 are executed, andthen the imaging procedure is ended. These processes are the same as theprocesses of Step S13 to Step S16 in FIG. 7, and hence descriptionthereof is omitted.

Note that, in Step S75 and Step S76, the intra-frame motion detectionand the inter-frame motion detection are performed on the basis of thedetection images so as to obtain the motion information item. Further,the processes of Step S75 and Step S76 are executed concurrently witheach other. In addition, in Step S77, the motion correction is performedwith respect to the input image, and then the output image is generated.

As described above, the endoscope system 11 performs the normal lightemission in the time period corresponding to the one field being firsthalf of one-frame time period so as to obtain the input image, andperforms the pulsed-light emissions in the time period corresponding tothe other one field being second half of the one-frame time period so asto obtain the detection image. Then, the endoscope system 11 detects themotion from the detection image, and generates the output image with useof the result of the detection and the input image. With this, themotion detection can be performed with higher accuracy and with a smalldelay.

Fourth Embodiment

<Motion Detection>

Further, when the time period in which the normal light emission isperformed to obtain the input image, and the time period in which thepulsed-light emissions are performed to obtain the detection image areseparated from each other, the normal light emission may be basicallyperformed, and the pulsed-light emissions may be temporarily performedwhen necessary.

In such a case, as shown in FIG. 13, for example, whether to perform thenormal light emission or to perform the pulsed-light emissions may beswitched on a frame-by-frame basis. In frames in each of which thepulsed-light emissions are performed, the input images may be obtainedby the interpolation process. Note that, in FIG. 13, componentscorresponding to those in the case of FIG. 2 are denoted by the samereference symbols, and description thereof is omitted as appropriate.Further, in FIG. 13, the abscissa axis represents time, and downwardarrows in FIG. 13 indicate timings of the light emissions by the lightsource device 23.

In the example shown in FIG. 13, the first exposure time periods in eachof which the white light beam is continuously output, and the secondexposure time periods in each of which the pulsed-light emission of thewhite light beam is performed a plurality of times are provided. Then,among the plurality of frames constituting the taken moving image,images of frames corresponding to the first exposure time periods areused as the input images, and images of frames corresponding to thesecond exposure time periods are used as the detection images.

In this way, in the endoscope system 11, ones of the frames of the takenmoving image are used as the input images, and other ones of the framesare used as the detection images. In the following, a sentence “inputimages and detection images are taken on a frame-by-frame basis” is alsoused.

In FIG. 13, in a time period T51 corresponding to a predetermined oneframe, as, for example, in the time period T41 in FIG. 11, the lightsource device 23 continuously emits the white light beam (performs thenormal light emission). This time period T51 is set as the time periodfor exposing the input image. In other words, when the time period T51has elapsed, the input image corresponding to one frame is obtained.

Further, in a time period T52 subsequent to the time period T51 andcorresponding to one frame, as in the time period T42 in FIG. 11, thelight source device 23 performs the pulsed-light emission twice so as totake the detection image. In other words, the time period T52 is set asthe exposure time period in which the pulsed-light emission is performedtwice, and an image corresponding to one frame, which is obtained inthis time period T52, is used as the detection image.

Respectively in four time periods subsequent to the time period T52,each of which corresponds to one frame, the light source device 23perform the normal light emissions of the white light beams so as totake the input images corresponding respectively to the frames. In atime period T53 subsequent thereto and corresponding to one frame, as inthe time period T52, the light source device 23 performs thepulsed-light emission twice so as to take the detection image.

In this way, in the example shown in FIG. 13, between the frames in eachof which the normal light emission is performed, the frames in each ofwhich the pulsed-light emissions are performed are partially inserted.Note that, intervals of the frames (exposure time periods) in each ofwhich the pulsed-light emissions are performed, the intervals beingpartially inserted between the frames in each of which the normal lightemission is performed, may be equal or may be unequal.

For example, when frames in each of which the pulsed-light emission isperformed a plurality of times are arranged at unequal intervals, thesignal processing circuit 32 may control the imaging unit 62 and thelight-source control unit 61 in response, for example, to operationinputs by a user such that the frames in each of which the pulsed-lightemissions are performed can be inserted at arbitrary timings. In otherwords, the signal processing circuit 32 may be capable of switchingwhether to take the input images or to take the detection images at thearbitrary timings.

In such a case, as, for example, in the time period T51, in the frame inwhich the normal light emission is performed, the output-imagegeneration unit 64 generates the output image from the taken input imagecorresponding to one frame.

Further, as, for example, in the time period T52, in the frame in whichthe pulsed-light emissions are performed, the motion detection unit 63calculates, as the motion information item, the motion vector from thedetection image by the same method as that in the first embodiment. Atthe time of calculating the direction of the motion, there may be used,for example, a detection image corresponding to closest one of previousframes in each of which the pulsed-light emissions are performed, theclosest one being closest to a processing-target frame in which thepulsed-light emissions are performed. Alternatively, there may be usedan input image corresponding to an adjacent frame in which the normallight emission is performed, the adjacent frame being temporallyadjacent to the processing-target frame in which the pulsed-lightemissions are performed.

Further, as, for example, in the time period T52, in the frame in whichthe pulsed-light emissions are performed, the input image is notobtained. Thus, in this frame, the input image is generated by theinterpolation process including motion compensation that is performed atleast from an input image corresponding to an immediately-precedingframe, and from the motion information item.

Specifically, the output-image generation unit 64 performs, for example,the motion compensation with respect to the input image of the framecorresponding to the time period T51 with use of the motion informationitem obtained in the frame corresponding to the time period T52, whichis supplied from the motion detection unit 63. With this, theoutput-image generation unit 64 generates an input image of the framecorresponding to the time period T52.

Note that, the input image of the frame corresponding to the time periodT52 may be generated with use of input images of the plurality ofadjacent frames that are preceding and subsequent to this frame. Forexample, the input image of the frame corresponding to the time periodT52 may generated by executing the interpolation process, specifically,by performing the motion compensation with use of the motion informationitem and from the input image of the frame corresponding to the timeperiod T51, and the input image of the frame immediately subsequent tothe frame corresponding to the time period T52.

Further, when the motion compensation is performed with respect to theinput images corresponding to the frames, a motion information item ofcorresponding one of the frames, or a motion information item of closestone of the previous frames, which is closest to the corresponding one ofthe frames, is used.

<Yet Another Imaging Procedure>

Next, a procedure that is executed in the endoscope system 11 at thetime when the frames in each of which the pulsed-light emissions areperformed are inserted as appropriate between the frames in each ofwhich the normal light emission is performed is described. In otherwords, in the following, with reference to the flowchart of FIG. 14, yetanother imaging procedure by the endoscope system 11 is described.

In Step S101, the light-source control unit 61 determines whether or nota processing-target current frame is a motion detection frame, that is,whether or not the processing-target current frame is the frame in whichthe pulsed-light emissions are performed.

In Step S101, when the processing-target current frame is determined tobe the motion detection frame, the procedure proceeds to Step S102.Then, processes of Step S102 to Step S105 are executed. With this, thedetection image is taken, and the motion information item is generated.Note that, the processes of these Step S102 to Step S105 are the same asthe processes of Step S73 to Step S76 in FIG. 12, and hence descriptionthereof is omitted.

In Step S106, the output-image generation unit 64 executes theinterpolation process on the basis of the motion information itemobtained from the processing-target current frame in Step S105, and ofthe input image supplied from the imaging unit 62. With this, theoutput-image generation unit 64 generates, as the output image, theinput image corresponding to the processing-target current frame.

In the interpolation process, for example, at least the input imagecorresponding to a frame immediately preceding the current frame isused. After the output image is obtained in this way, the procedureproceeds to Step S110.

Further, in Step S101, when the processing-target current frame isdetermined not to be the motion detection frame, that is, determined tobe the frame in which the normal light emission is performed, theprocedure proceed to Step S107.

Then, processes of Step S107 to Step S109 are executed, and the outputimage corresponding to the processing-target current frame is generated.Note that, the processes of these Step S107 to Step S109 are the same asthe processes of Step S71, Step S72, and Step S77 in FIG. 12, and hencedescription thereof is omitted.

Note that, in Step S109, the motion correction is performed with use ofa motion information item obtained from temporally closest one of theframes preceding the processing-target current frame, in each of whichthe pulsed-light emissions are performed, that is, with use of themotion information item obtained most lately by the process of StepS105.

After the output image is obtained by performing the motion correctionin Step S109, the procedure proceeds to Step S110.

When the output image is obtained by executing the process of Step S109or Step S106, in Step S110, the output-image generation unit 64 outputsthe obtained output image to the recorder 65 or the monitor 26. Then,the imaging procedure is ended. With this, the output image is recordedin the recorder 65, or the output image is displayed on the monitor 26.

As described above, the endoscope system 11 performs the normal lightemission in each of the normal frames so as to obtain the input images,and performs the pulsed-light emissions in each of the frames insertedas appropriate so as to obtain the detection images. Then, the endoscopesystem 11 detects the motion from the detection images, and generatesthe output image with use of the result of the detection and the inputimages. With this, the motion detection can be performed with higheraccuracy and with a small delay.

Fifth Embodiment

<Motion Detection>

Note that, although whether to perform the normal light emissions or toperform the pulsed-light emissions is switched on the frame-by-framebasis or the field-by-field basis in the embodiments describedhereinabove, when, for example, the light beams to be output by thenormal light emissions, and the light beams to be output by thepulsed-light emissions have different wavelength components, these lightbeams can be emitted together.

Specifically, as shown in FIG. 15, for example, in the time periodcorresponding to each of the frames, the normal light emission can becontinuously performed, and the pulsed-light emission can be performedtwice per frame. Note that, in FIG. 15, the abscissa axis representstime, and downward arrows in FIG. 15 indicate timings of the lightemissions by the light source device 23.

In this case, for example, the light source device 23 includes the laserlight source that performs the normal light emissions so as to outputthe white light beams as the illumination light beams, that is, the RGBlaser-light source constituted by the “R” light source, the “G” lightsource, and the “B” light source. The light source device 23 alsoincludes an infrared-laser light source that performs the pulsed-lightemissions so as to output infrared rays for the motion detection.

The infrared-laser light source is a laser light source capable ofpulsed-light emissions of, for example, infrared rays having awavelength of 850 nm. Further, infrared components are not contained inthe white light beams as the illumination light beams. The laser lightsource that outputs such white light beams is not limited to the RGBlaser-light source, and may be other light sources such as a xenon lightsource and an LED light source.

In the example shown in FIG. 15, for example, a time period T61 is atime period of one frame corresponding to the input image and thedetection image. During this time period T61, the normal light emissionof the white light beam is continuously performed. Further, in the timeperiod T61, the pulsed-light emission of the infrared ray is alsoperformed twice. In this example, the pulsed-light emissions areperformed respectively at two time points, that is, a time point t61 anda time point t62. With this, the infrared rays are applied to thephotographic subject.

When the normal light emission of the white light beam and thepulsed-light emissions of the infrared rays are performed in one-frametime period in this way, on the imaging unit 62 side, that is, on thecamera head 22 side, an optical system that splits these white lightbeam and infrared rays from each other, and sensors (imaging units) thatrespectively receive the white light beam and the infrared rays areneeded.

Thus, when the normal light emission of the white light beam and thepulsed-light emissions of the infrared rays are performed in theone-frame time period, more specifically, the imaging unit 62 isconfigured, for example, as shown in FIG. 16. Note that, in FIG. 16,components corresponding to those in the case of FIG. 2 are denoted bythe same reference symbols, and description thereof is omitted asappropriate.

In the example shown in FIG. 16, the imaging unit 62 includes asplitting element 101, a mirror 102, a visible-light sensor 103, and aninfrared sensor 104.

In this example, in the time period corresponding to each of the frames,the reflected light beam of the white light beam by the normal lightemission, and reflected rays of the infrared rays by the pulsed-lightemissions from the photographic subject enter the splitting element 101.

The splitting element 101, which is constituted, for example, by aprism, a dichroic mirror, or a half mirror, optically splits theincident reflected-light beams from the photographic subject. Further,the mirror 102, which is constituted, for example, by a total reflectionmirror, reflects and inputs the incident light beams from the splittingelement 101 to the infrared sensor 104.

The visible-light sensor 103, which is constituted by an image sensorincluding a plurality of pixels that receive and photoelectricallyconvert visible light beams, takes the input images by receiving andphotoelectrically converting the incident white light beams from thesplitting element 101, and supplies the obtained input images to theoutput-image generation unit 64. This visible-light sensor 103 functionsas the imaging unit that takes the input images for obtaining theobservational output images.

The pixels of the visible-light sensor 103 are each provided with aninfrared cut filter that blocks the infrared rays. With this,visible-light components of the light beams that have entered thevisible-light sensor 103, that is, only the components of the whitelight beams are received by photodiodes of the pixels.

Further, the infrared sensor 104, which is constituted by an imagesensor including a plurality of pixels that receives andphotoelectrically converts the infrared rays, takes the detection imagesby receiving and photoelectrically converting the incident infrared raysfrom the mirror 102, and supplies the obtained detection images to themotion detection unit 63. This infrared sensor 104 functions as theimaging unit that takes the detection images to be used for the motiondetection.

The pixels of the infrared sensor 104 are each provided with avisible-light cut filter that blocks the visible light beams. With this,only the components of the infrared rays among the light beams that haveentered the infrared sensor 104 are received by photodiodes of thepixels.

For example, when the splitting element 101 is constituted by the prismor the dichroic mirror, the splitting element 101 transmits and inputs,to the visible-light sensor 103, the components of the white light beamsamong the incident reflected-light beams from the photographic subject,that is, the visible-light components. In addition, the splittingelement 101 reflects the components of the infrared rays among theincident reflected-light beams from the photographic subject. Then, theinfrared rays, which are reflected by the splitting element 101 andenter the mirror 102, are reflected by the mirror 102, and then enterthe infrared sensor 104.

With this, visible-light input images are obtained by the visible-lightsensor 103, and infrared detection images are obtained by the infraredsensor 104.

Further, when the splitting element 101 is constituted by the halfmirror, the splitting element 101 transmits and inputs, to thevisible-light sensor 103, ones of the incident light beams from thephotographic subject. In addition, the splitting element 101 reflectsrest of the light beams, that is, other ones of the incident lightbeams. At this time, the rest of the light beams, which is reflected bythe splitting element 101 and enter the mirror 102, is reflected by themirror 102, and then enter the infrared sensor 104.

In this case, the ones of the light beams, which enter the visible-lightsensor 103, include the white-light components and the infraredcomponents. However, the infrared rays are blocked by the infrared cutfilter provided to each of the pixels of the visible-light sensor 103,and only the white light beams are received by the visible-light sensor103. With this, the visible-light input images are obtained by thevisible-light sensor 103.

Further, the other ones of the light beams, which enter the infraredsensor 104, also include the white-light components and the infraredcomponents. However, the visible light beams are blocked by thevisible-light cut filter provided to each of the pixels of the infraredsensor 104, and only the infrared rays are received by the infraredsensor 104. With this, the infrared detection images are obtained by theinfrared sensor 104.

<Yet Another Imaging Procedure>

Next, a procedure that is executed in the endoscope system 11 at thetime when the normal light emission and the pulsed-light emissions areperformed in one-frame time period is described. In other words, in thefollowing, with reference to the flowchart of FIG. 17, yet anotherimaging procedure by the endoscope system 11 is described.

In Step S141, the light source device 23 performs not only the normallight emission so as to continuously apply the white light beam as theillumination light beam to the photographic subject during the one-frametime period, but also the pulsed-light emission twice at predeterminedtimings in the one-frame time period so as to apply the infrared rays tothe photographic subject.

Specifically, the light-source control unit 61 controls the RGB-laserlight source as the light source device 23 such that the RGB-laser lightsource continuously outputs the white light beam during the exposuretime period of the input image. In addition, the light-source controlunit 61 controls the infrared-laser light source as the light sourcedevice 23 such that the infrared-laser light source outputs the infraredrays each having the wavelength different from that of the white lightbeam by performing the pulsed-light emission twice during the exposuretime period of the detection image. In this procedure, the exposure timeperiod of the input image and the exposure time period of the detectionimage are basically the same time period. However, there is noparticular problem as long as the exposure time period of the detectionimage, that is, the time period in which the pulsed-light emission isperformed twice includes at least a part of the exposure time period ofthe input image, in which the normal light emission by the RGB-laserlight source is performed.

By such light-emission control, at each of the time points in theone-frame time period, only the white light beam or both the white lightbeam and the infrared ray are applied to the photographic subject. Thereflected light beams generated by the application of the white lightbeam and the infrared rays to the photographic subject enter thesplitting element 101 of the imaging unit 62. For example, the whitelight beam among the reflected light beams transmits through thesplitting element 101, and then enters the visible-light sensor 103.Further, the infrared rays among the reflected light beams are reflectedby the splitting element 101 and the mirror 102, and then enter theinfrared sensor 104.

In Step S142, the visible-light sensor 103 of the imaging unit 62 takesthe input image by receiving and photoelectrically converting theincident white light beam, and supplies the obtained input image to theoutput-image generation unit 64.

In Step S143, the infrared sensor 104 of the imaging unit 62 takes thedetection image by receiving and photoelectrically converting theinfrared rays, and supplies the obtained detection image to the motiondetection unit 63. Note that, the processes of these Step S142 and StepS143 are executed simultaneously with each other.

After the input image and the detection image are obtained, processes ofStep S144 to Step S147 are executed, and then the imaging procedure isended. These processes are the same as the processes of Step S75 to StepS78 in FIG. 12, and hence description thereof is omitted. Note that, theprocesses of Step S144 and Step S145 are executed concurrently with eachother, and that the motion correction that is performed with respect tothe input image in Step S146 is on the basis of the motion informationitem obtained from the detection image.

As described above, the endoscope system 11 performs the normal lightemission and the pulsed-light emissions together in one-frame timeperiod so as to obtain the input image and the detection image. Then,the endoscope system 11 detects the motion from the detection image, andgenerates the output image with use of the result of the detection andthe input image. With this, the motion detection can be performed withhigher accuracy and with a small delay.

Sixth Embodiment

<Another Functional Configuration Example of Endoscope System>

Incidentally, the endoscope system 11 to which the present technology isapplied is advantageous also at a time of performing vocal-cordobservation.

Generally, vocal cords vibrate at a frequency of several hundred Hz, andhence still images of the vocal cord parts cannot be acquired by normalobservation. In view of such circumstances, there has been devisedlaryngo-stroboscopy including applying illumination light beams from astroboscopic light source in accordance with vibration periods of thevocal cords such that the vocal cords to be observed seem still.

However, in the laryngo-stroboscopy, in order to detect the vibrationperiods of the vocal cords, voice produced by a participant needs to beacquired, resulting in complications of the endoscope system. Inaddition, the participant needs to continue to produce the voice at acertain pitch for a certain time period.

As a countermeasure, in the endoscope system 11, the illumination lightbeams are applied by the pulsed-light emissions to the vocal cords asthe photographic subject. With this, the vibration periods of the vocalcords, that is, the frequency of the vibration is detected on the basisof the input images obtained by capturing the vocal cords. In this way,the still images of the vocal cords are obtained as the output images.

In this case, in the same periods as the periods of the detectedvibration of the vocal cords, that is, in synchronization with thevibration of the vocal cords, the pulsed-light emissions, that is,stroboscopic lighting by the light source device 23 is performed. Withthis, the still images of the vocal cords can be more easily obtained.

The endoscope system 11 at a time of performing such vocal-cordobservation is functionally configured, for example, as shown in FIG.18. Note that, in FIG. 18, components corresponding to those in the caseof FIG. 2 are denoted by the same reference symbols, and descriptionthereof is omitted as appropriate.

In the example shown in FIG. 18, the endoscope system 11 includes thelight-source control unit 61, the light source device 23, the imagingunit 62, a period detection unit 141, and an output-image generationunit 142.

Further, in this example, the light-source control unit 61 corresponds,for example, to the light-source control device 31 shown in FIG. 1, andthe imaging unit 62 corresponds to the camera head 22 shown in FIG. 1.Further, the detection unit 33 in FIG. 1 functions as the perioddetection unit 141, and the signal processing circuit 32 in FIG. 1functions as the output-image generation unit 142.

The light source device 23 performs the pulsed-light emissions inpredetermined periods under the control by the light-source control unit61 so as to apply pulsed light beams as the illumination light beams tothe photographic subject. Specifically, the controlled light-sourcedevice 23 performs the pulsed-light emission a plurality of times inone-frame time period corresponding to the input image.

When the illumination light beams are applied to the vocal cords as thephotographic subject, these illumination light beams turn into thereflected light beams by being reflected by the vocal cords, and thenenter the imaging unit 62.

The imaging unit 62 takes the input images depicting the vocal cords asthe photographic subject by receiving and photoelectrically convertingthe reflected light beams from the vocal cords. Then, the imaging unit62 supplies the obtained input images to the period detection unit 141and the output-image generation unit 142.

The period detection unit 141 detects, on the basis of the input imagesat respective time points, which are supplied from the imaging unit 62,the vibration periods of the vocal cords, that is, the frequency of thevibration of the vocal cords as motions of the photographic subject.Then, the period detection unit 141 supplies results of the detection asmotion information items indicating the detection results of the motionsof the vocal cords to the light-source control unit 61.

For example, the light source device 23 performs the pulsed-lightemission the plurality of times in the one-frame time periodcorresponding to each of the input images, and the vocal cords alsovibrate a plurality of times in the one-frame time period. Thus, themultiple exposure is performed at the time of taking each of the inputimages, and hence a plurality of images of the same photographic subjectare contained in each of the input images.

Thus, when the periods of the pulsed-light emissions by the light sourcedevice 23, that is, a flashing frequency at which the illumination lightbeams are turned ON/OFF and the frequency of the vibration of the vocalcords are the same as each other, as illustrated in FIG. 19, forexample, a high-contrast input image P41 is obtained.

The input image P41 illustrated in FIG. 19 is a high-contrast unblurredimage of the vocal cords being the photographic subject. In other words,in the input image P41 to be obtained, the vocal cords as thephotographic subject are depicted substantially in a stationary state.

This is because, when the frequency of the pulsed-light emissions andthe frequency of the vibration of the vocal cords are the same as eachother, images of the vocal cords at respective time points in the inputimage P41, that is, at times of respective ones of the pulsed-lightemissions are superimposed on each other. For this reason, the image ofthe photographic subject does not blur.

In contrast, when the frequency of the pulsed-light emissions by thelight source device 23 and the frequency of the vibration of the vocalcords are shifted from each other, positions of the images of the vocalcords (photographic subject) in the input image are also shifted fromeach other. Thus, for example, a low-contrast image such as an inputimage P42 in FIG. 19 is obtained.

As long as the frequency of the pulsed-light emissions and the frequencyof the vibration of the vocal cords are different from each other, whenthe pulsed-light emission is performed the plurality of times in theone-frame time period, the positions of the vocal cords at the timingsof the respective pulsed-light emissions are different from each other.Thus, positions of the images of the vocal cords in the input image P42are also different from each other. In other words, a low-contrastblurred image including multiple images is obtained as the input imageP42.

Referring back to the description with reference to FIG. 18, the perioddetection unit 141 detects the motions of the vocal cords, that is, thefrequency of the vocal cords by utilizing a relationship between adegree of such frequency matching and the contrasts of the input images.

Specifically, for example, the period detection unit 141 calculatescontrast evaluation values each representing a degree of a contrast ofthe input image obtained at each of the frequencies of the pulsed-lightemissions, and detects a peak value of the contrast evaluation valuesfrom these contrast evaluation values obtained from the frequencies.Then, the period detection unit 141 determines one of the frequencies ofthe pulsed-light emissions, which corresponds to the peak value of thecontrast evaluation values, as the frequency of the vibration of thevocal cords. The period detection unit 141 supplies an information itemindicating the frequency or periods thereof as the motion informationitem being the detection result of the motion of the vocal cords as thephotographic subject to the light-source control unit 61. Thelight-source control unit 61 changes the frequency, that is, the periodsof the pulsed-light emission on the basis of the motion information itemsupplied from the period detection unit 141, and causes the light sourcedevice 23 to perform the light-emitting operation in the changedperiods.

In particular, in the endoscope system 11, the stroboscopic lighting,that is, the pulsed-light emissions by the light source device 23 areperformed in the plurality of different periods while gradually changingthe frequency (periods) of the pulsed-light emissions. Then, thefrequency of the vocal cords is detected from the contrast evaluationvalue obtained at each of the plurality of different frequencies(periods). Such a method of controlling the pulsed-light emissions isthe same control method as contrast-type automatic focusing controlincluding moving a lens so as to detect a lens position whichcorresponds to the peak value of the evaluation values of the contrasts.When this control method is employed, the frequency of the vibration ofthe vocal cords can be detected with high accuracy.

After the frequency of the vocal cords is detected as described above,the endoscope system 11 takes input images by performing thepulsed-light emissions at the same frequency as that indicated by themotion information item, that is, as the detected frequency. In thisway, unblurred still output images of the vocal cords are obtained.

In other words, after the motion of the vocal cords as the photographicsubject is detected, the input images taken by the imaging unit 62 aresupplied to the output-image generation unit 142. The output-imagegeneration unit 142 generates the output images by executingpredetermined processes such as a gain control process on the inputimages supplied from the imaging unit 62, and then outputs these outputimages. The output images obtained in this way are the unblurred stillimages of the vocal cords.

<Yet Another Imaging Procedure>

Next, a procedure that is executed by the endoscope system 11 at thetime when the vocal-cord observation is performed is described. In otherwords, in the following, with reference to the flowchart of FIG. 20, yetanother imaging procedure by the endoscope system 11 is described.

In Step S181, the light-source control unit 61 determines a newfrequency of the pulsed-light emissions on the basis of previousfrequencies of the pulsed-light emissions.

For example, when first light-emissions are performed, a preset certainfrequency is determined as the frequency of the pulsed-light emissions.Further, for example, when the motion information item has not yet beensupplied despite previous gradual increase in frequency of thepulsed-light emissions, in other words, when the frequency of the vocalcords has not yet been detected, a much higher frequency is determinedas the new frequency of the pulsed-light emissions.

In Step S182, the light source device 23 performs the pulsed-lightemissions. Specifically, the light-source control unit 61 controls thelight source device 23 such that the light source device 23 performs thepulsed-light emissions at the frequency determined in Step S181. Thelight source device 23 performs the pulsed-light emissions at thepredetermined frequency under the control by the light-source controlunit 61. With this, in the time period of one frame corresponding to theinput image, the illumination light beams are applied periodically tothe vocal cords being the photographic subject. These illumination lightbeams turn into the reflected light beams by being reflected by thevocal cords, and then enter the imaging unit 62. Note that, at thistime, the participant continuously produces the voice at the certainpitch.

In Step S183, the imaging unit 62 takes, for example, the input imageseach corresponding to one frame by receiving and photoelectricallyconverting the incident reflected-light beams from the vocal cords. Theimaging unit 62 supplies the taken input images to the period detectionunit 141.

In Step S184, the period detection unit 141 detects the motion of thevocal cords being the photographic subject on the basis of the inputimages supplied from the imaging unit 62.

Specifically, the period detection unit 141 calculates the contrastevaluation value on the basis of the input images, and detects the peakvalue of the contrast evaluation values from previously-calculatedcontrast evaluation values.

In Step S185, on the basis of the detection result of the motion in StepS184, that is, on the basis of the detection result of the peak value ofthe contrast evaluation values, the period detection unit 141 determineswhether or not the frequency of the vocal cords has been detected. Forexample, when the peak value of the contrast evaluation values has beendetected, it is determined that the frequency of the vocal cords hasbeen detected.

When it is determined in Step S185 that the frequency of the vocal cordshas not yet been detected, the procedure returns to Step S181, and theabove-described processes are repeated. In this case, imaging isperformed at a different pulsed-light emission frequency, and thecontrast evaluation values are calculated.

In contrast, it is determined in Step S185 that the frequency of thevocal cords has been detected, the period detection unit 141 determinesthe frequency of the pulsed-light emissions, at which the peak value ofthe contrast evaluation values is obtained, as the frequency of thevocal cords, more specifically, the frequency of the vibration of thevocal cords. Then, the period detection unit 141 supplies theinformation item indicating the frequency of the vibration of the vocalcords as the motion information item indicating the detection result ofthe motion to the light-source control unit 61. Next, the procedureproceeds to Step S186. Note that, the period detection unit 141 hasalready grasped the frequency of the pulsed-light emissions atrespective time points by the light source device 23.

In Step S186, the light-source control unit 61 sets a frequency ofsubsequent pulsed-light emissions to the frequency of the vocal cords,which is indicated by the motion information item, and controls thelight source device 23 such that the light source device 23 performs thesubsequent pulsed-light emissions at this frequency.

Then, in Step S187, the light source device 23 performs the pulsed-lightemissions under the control by the light-source control unit 61. Withthis, the pulsed-light emissions of the illumination light beams at thesame frequency as the frequency of the vibration of the vocal cords.

In Step S188, the imaging unit 62 takes the input images by receivingand photoelectrically converting the incident reflected-light beams fromthe vocal cords, and supplies the obtained input images to theoutput-image generation unit 142.

In Step S189, the output-image generation unit 142 generates the outputimages on the basis of the input images supplied from the imaging unit62, and then outputs these output images. Then, the imaging procedure isended. For example, the output images are generated by executingnecessary processes such as white-balance adjustment and the gainadjustment on the input images as appropriate. The output images outputfrom the output-image generation unit 142 are, for example, displayed onthe monitor 26, or recorded in the recorder.

As described above, the endoscope system 11 performs the pulsed-lightemission a plurality of times in one-frame time period while changingthe frequency, and detects the frequency of the vibration of the vocalcords by calculating the contrast evaluation values of the taken inputimages. Then, the endoscope system 11 takes the input images byperforming the pulsed-light emissions at the detected frequency. Withthis, the motionless still output images of the vocal cords areobtained.

By performing the pulsed-light emissions while changing the frequency,and by calculating the contrast evaluation values, the motion of thevocal cords being the photographic subject can be detected more easilyand with higher accuracy.

In particular, in the endoscope system 11, it is unnecessary to collectthe voice produced by the participant, and the frequency of thevibration of the vocal cords is detected from the contrasts of theimages. Thus, the frequency of the vibration of the vocal cords can bedetected easily with a simple configuration and with high accuracy.

In addition, in the endoscope system 11, even when the participantfluctuates the pitch of the voice halfway, by detecting again thefrequency of the vibration of the vocal cords and adjusting thefrequency of the pulsed-light emissions while taking the input images,it is possible to follow the fluctuation of the pitch of the voice ofthe participant.

Incidentally, in the example described hereinabove, the output images ofthe vocal cords in the stationary state are generated by setting thefrequency of the pulsed-light emissions the same as the frequency of thevibration of the vocal cords.

However, when the frequency of the pulsed-light emissions is slightlyshifted from the frequency of the vibration of the vocal cords, as shownin FIG. 21, for example, a moving image in which the vocal cords appearto vibrate slowly, that is, a moving image in which the vibration of thevocal cords appears to be reproduced in slow motion can be obtained asthe output images. Note that, in FIG. 21, the abscissa axis representstime, and the ordinate axis represents positions of the vocal cords.

In the example shown in FIG. 21, a curve L11 represents positions of thevocal cords at respective time points. Further, circles in FIG. 21represent timings of the pulsed-light emissions, and a curve L12represents positions of the vocal cords at respective time points, whichare observed in the output images.

In this example, as understood from the curve L11, the vocal cords ofthe participant vibrate in certain periods, that is, at a certainfrequency. Meanwhile, as understood from positions of the circles, inthe endoscope system 11, the pulsed-light emissions are performed at acertain frequency different from the frequency of the vocal cords.

With this, the positions of the vocal cords in the input imagescorrespond to the positions of the circles on the curve L11. As aresult, the positions of the vocal cords at the respective time points,which are observed in the output images being the moving image,corresponds to the positions represented by the curve L12. Thus, thecurve L12 can also be interpreted as a curve representing periods(frequency) of seeming vibration of the vocal cords in the outputimages.

When the frequency of the pulsed-light emissions is slightly shiftedfrom the frequency of the vibration of the vocal cords as describedabove, the input images are exposed only at the timings represented bythe circles. Thus, the output images in which the vocal cords, which areto vibrate in the periods represented by the curve L11, appear tovibrate in the periods represented by the curve L12 can be obtained. Inother words, the moving image reproducing substantially in slow motionthe motions of the vocal cords can be obtained as the output images.

In this case, it is only necessary to set, in Step S186 of the yetanother imaging procedure described with reference to FIG. 20, afrequency shifted by a predetermined value from the detected frequencyof the vocal cords as the frequency of the pulsed-light emissions. Atthis time, by properly controlling the frequency of the pulsed-lightemissions, a rate of the motions of the vocal cords, which arereproduced in the output images, can be adjusted to a desired rate.

When the vocal-cord observation, that is, the laryngo-stroboscopy isperformed with use of the endoscope system 11 as described above,without use of the configuration for collecting the voice of theparticipant, and detecting a frequency of the voice, still vocal cords,and vocal cords that slowly move can be observed. Further, theobservation that enables even the pitch fluctuated halfway by theparticipant to be followed can be performed.

In addition, the present technology described hereinabove isadvantageous also in detection of motions much more rapid than timeintervals on a frame-by-frame basis, such as detection of a pulse wavevelocity of a blood vessel, and motion detection by laser speckleimaging.

<Configuration Example of Endoscopic Surgical System>

Further, the present technology is applicable, for example, also to anendoscopic surgical system for performing endoscopic surgery with use ofthe endoscope system 11 shown in FIG. 1. In such a case, the endoscopicsurgical system is configured, for example, as shown in FIG. 22. Inother words, FIG. 22 is a diagram showing an example of a schematicconfiguration of an endoscopic surgical system 500 to which thetechnology according to the present disclosure is applicable.

FIG. 22 shows a state in which an operator (doctor) 567 performs surgeryon a patient 571 on a patient bed 569 with use of the endoscopicsurgical system 500. As shown in FIG. 22, the endoscopic surgical system500 is constituted by an endoscope 501, other surgical instruments 517,a support arm device 527 that supports the endoscope 501, and a cart 537to which various devices for the endoscopic surgery are mounted.

In the endoscopic surgery, an abdominal-wall incision is not performed.Instead, abdominal-wall puncture with a plurality of barrel-like openinginstruments called trocars 525 a to 525 d is performed. Then, throughthe trocars 525 a to 525 d, a lens barrel 503 of the endoscope 501, andthe other surgical instruments 517 are inserted into a body cavity ofthe patient 571. In the example shown in FIG. 22, as the other surgicalinstruments 517, an insufflation tube 519, an energy treatmentinstrument 521, and forceps 523 are inserted into the body cavity of thepatient 571. Further, the energy treatment instrument 521 is a treatmentinstrument that performs, for example, incision and dissection of tissueor blood-vessel sealing with high-frequency current or ultrasonicvibration. Note that, the surgical instruments 517 shown in FIG. 22 aremerely examples, and various surgical instruments to be generally usedin the endoscopic surgery, such as tweezers and a retractor, may be usedas the surgical instruments 517.

Images of surgical parts in the body cavity of the patient 571 are takenwith the endoscope 501, and displayed on a display device 541. Whilechecking the images of the surgical parts in real time, which aredisplayed on the display device 541, the operator 567 performs treatmentsuch as resection of affected parts with use of the energy treatmentinstrument 521 and the forceps 523. Note that, although not shown,during the surgery, the insufflation tube 519, the energy treatmentinstrument 521, and the forceps 523 are supported by the operator 567or, for example, by an assistant.

(Support Arm Device)

The support arm device 527 includes an arm portion 531 that extends froma base portion 529. In the example shown in FIG. 22, the arm portion531, which is constituted by joint portions 533 a, 533 b, and 533 c, andlinks 535 a and 535 b, is driven under control by an arm control device545. The endoscope 501 is controlled in position and posture by beingsupported by the arm portion 531. With this, the endoscope 501 is stablyfixed in position.

(Endoscope)

The endoscope 501 is constituted by the lens barrel 503, which isinserted into the body cavity of the patient 571 in a region over apredetermined length from its distal end, and by a camera head 505 thatis connected to a proximal end of the lens barrel 503. The endoscope 501in the example shown in FIG. 22 is constituted as what is called a rigidscope, that is, the lens barrel 503 thereof is rigid. However, theendoscope 501 may be constituted as what is called a flexible scope,that is, the lens barrel 503 may be flexible.

An opening portion into which an objective lens is fitted is provided atthe distal end of the lens barrel 503. A light source device 543 isconnected to the endoscope 501. Light beams generated by the lightsource device 543 are guided to the distal end of the lens barrel 503 bya light guide extended in the lens barrel, and are applied through theobjective lens to an observation target in the body cavity of thepatient 571. Note that, the endoscope 501 may be a forward-viewingendoscope, a forward-oblique viewing endoscope, or a side-viewingendoscope.

An optical system and an imaging element are provided in the camera head505, and reflected light beams (observation light beams) from theobservation target are converged to the imaging element by the opticalsystem. The observation light beams are photoelectrically converted bythe imaging element such that electrical signals corresponding to theobservation light beams, that is, image signals corresponding toobservation images are generated. These image signals are transmitted asRAW data items to a CCU (Camera Control Unit) 539. Note that, a functionto adjust a magnification and a focal length by driving the opticalsystem as appropriate is provided to the camera head 505.

For example, in order to perform stereoscopic viewing (three-dimensionalrepresentation), a plurality of imaging elements may be provided in thecamera head 505. In this case, in the lens barrel 503, in order that theobservation light beams are guided to each of the plurality of imagingelements, a plurality of relay optical systems are provided.

(Various Devices that are Mounted to Cart)

The CCU 539, which is constituted, for example, by a CPU (CentralProcessing Unit) and a GPU (Graphics Processing Unit), collectivelycontrols the operations of the endoscope 501 and the display device 541.Specifically, the CCU 539 executes, on the image signals received fromthe camera head 505, various image processes for displaying images inresponse to the image signals, such as a development process (demosaicprocess). The CCU 539 provides, to the display device 541, the imagesignals that have been subjected to the image processes. Further, theCCU 539 controls and drives the camera head 505 by transmitting controlsignals thereto. These control signals include information items ofimaging conditions such as the magnification and the focal length.

Under the control by the CCU 539, the display device 541 displays imagesin response to the image signals that have been subjected to the imageprocesses by the CCU 539. When the endoscope 501 corresponds, forexample, to high-resolution imaging such as 4K (3,840 horizontalpixels×2,160 vertical pixels) or 8K (7,680 horizontal pixels×4,320vertical pixels), and/or to the three-dimensional representation, adisplay device capable of high-resolution representation, and/or adisplay device capable of the three-dimensional representation can beused correspondingly thereto respectively as the display device 541.When the display device 541 to be used corresponds to thehigh-resolution imaging such as 4K or 8K, and has a size of 55 inches ormore, a greater sense of immersion can be obtained. Further, dependingon uses, the display device 541 to be provided may include a pluralityof display devices 541 having different resolutions and sizes.

The light source device 543, which is constituted, for example, by alight source such as an LED, supplies the illumination light beams atthe time of imaging the surgical parts to the endoscope 501.

The arm control device 545, which is constituted, for example, by aprocessor such as the CPU, is operated in accordance with apredetermined program. With this, the arm portion 531 of the support armdevice 527 is controlled and driven in accordance with a predeterminedcontrol method.

An input device 547 is an input/output interface with respect to theendoscopic surgical system 500. A user can input, via the input device547, various information items and instructions to the endoscopicsurgical system 500. For example, the user inputs, via the input device547, various information items of the surgery, such as physicalinformation items of the patient, and information items of a surgicalprocedure. Further, for example, the user inputs, via the input device547, instructions such as an instruction to drive the arm portion 531,an instruction to change the conditions of imaging (such as a type ofthe illumination light beams, the magnification, and the focal length)by the endoscope 501, and an instruction to drive the energy treatmentinstrument 521.

Types of the input device 547 are not limited. The input device 547 maybe various known input devices. For example, a mouse, a keyboard, atouchscreen, a switch, a foot switch 557, and/or a lever is applicableto the input device 547. When the touchscreen is used as the inputdevice 547, this touchscreen may be provided on a display surface on thedisplay device 541.

Alternatively, the input device 547 may be a device to be worn by theuser, such as an eyeglass-type wearable device or an HMD (Head MountedDisplay). Various inputs are performed by detection with these devices,specifically, by gestures or lines of sight of the user. Further, theinput device 547 includes a camera capable of detecting a motion of theuser such that the various inputs are performed by detection from avideo taken by the camera, specifically, by the gestures or the lines ofsight of the eyes of the user. In addition, the input device 547includes a microphone capable of collecting voice of the user such thatthe various inputs are performed by the voice via the microphone. Whenthe input device 547 is configured to be capable of enabling the variousinformation items to be input in a hands-free manner in this way, theuser (such as operator 567) who is in a clean area, in particular, isenabled to operate devices in a dirty area in the hands-free manner.Further, the user is enabled to operate these devices without releasinga surgical instrument in his/her hand. Thus, convenience of the user isincreased.

A treatment-instrument control device 549 controls and drives the energytreatment instrument 521 for, for example, cauterization and theincision of tissue or the blood-vessel sealing. An insufflation device551 feeds a gas into the body cavity of the patient 571 through theinsufflation tube 519 so as to insufflate the body cavity for purposesof securing a field of vision of the endoscope 501 and securing aworking space for the operator. A recorder 553 is a device capable ofrecording the various information items of the surgery. A printer 555 isa device capable of printing the various information items of thesurgery into various forms such as a text, an image, or a graph.

Further, there are correspondences as follows between the endoscopicsurgical system 500 shown in FIG. 22 and the endoscope system 11 shownin FIG. 11. Specifically, the lens barrel 503 corresponds to the scope21, and the camera head 505 corresponds to the camera head 22. Further,the light source device 543 corresponds to the light source device 23,the camera control unit 539 corresponds to the camera control unit 24,the input device 547 corresponds to the operation input device 25, andthe display device 541 corresponds to the monitor 26.

Hereinabove, an example of the endoscopic surgical system 500 to whichthe technology according to the present disclosure is applicable isdescribed. Note that, the system to which the technology according tothe present disclosure is applicable is not limited to the exampledescribed hereinabove of the endoscopic surgical system 500. Forexample, the technology according to the present disclosure may beapplied to an inspection flexible-endoscope system or a microscopicoperation system.

<Configuration Example of Computer>

Incidentally, the above-described series of processes may be executed byhardware or by software. At a time of executing the series of processesby the software, programs of the software are installed in the computer.Examples of the computer include a computer incorporated in dedicatedhardware, and a general-purpose computer capable of exerting variousfunctions in accordance with various programs installed therein.

FIG. 23 is a block diagram showing a configuration example of thehardware of a computer that executes the above-described series ofprocesses in accordance with the programs.

In the computer, a CPU 601, a ROM (Read Only Memory) 602, and a RAM(Random Access Memory) 603 are connected to each other via a bus 604.

An input/output interface 605 is also connected to the bus 604. An inputunit 606, an output unit 607, a recording unit 608, a communication unit609, and a drive 610 are connected to the input/output interface 605.

The input unit 606 includes a keyboard, a mouse, a microphone, and animaging element. The output unit 607 includes a display and a speaker.The recording unit 608 includes a hard disk and a nonvolatile memory.The communication unit 609 includes a network interface. The drive 610drives removable recording media 611 such as a magnetic disk, an opticaldisk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, for example, the CPU 601loads a program recorded in the recording unit 608 to the RAM 603 viathe input/output interface 605 and the bus 604, and executes theprogram. In this way, the above-described series of processes isexecuted.

The program to be executed by the computer (CPU 601) may be provided,for example, by being recorded in the removable recording medium 611 ina form of a packaged medium or the like. Alternatively, the program maybe provided via wired or wireless transmission media such as a localarea network, the Internet, and digital satellite broadcasting.

In the computer, the program may be installed into the recording unit608 via the input/output interface 605 from the removable recordingmedium 611 loaded to the drive 610. Alternatively, the program may beinstalled into the recording unit 608 by being received by thecommunication unit 609 via wired and wireless transmission media. Stillalternatively, the program may be pre-installed in the ROM 602 or therecording unit 608.

Note that, the program to be executed by the computer may be programsfor executing the processes in time series in the order describedherein, or may be programs for executing the processes parallel to eachother or at necessary timings, for example, when being called.

In addition, the embodiments of the present technology are not limitedto the above-described embodiments, and various modifications may bemade thereto without departing from the essence of the presenttechnology.

For example, the present technology may have a configuration of cloudcomputing in which one function is shared by a plurality of devices viaa network and processed in cooperation with each other.

Further, Steps described above with reference to the flowcharts may beexecuted by a single device, or may be executed by a plurality ofdevices in a shared manner.

Still further, when a plurality of processes are contained in a singleStep, the plurality of processes contained in the single Step may beexecuted by a single device, or may be executed by a plurality ofdevices in a shared manner.

Yet further, the advantages described herein are merely examples, andhence are not limited thereto. Thus, other advantages may be obtained.

Yet further, the present technology may also employ the followingconfigurations.

(1)

An image processing device, including:

a light-source control unit that controls a light source such that thelight source performs a pulsed-light emission a plurality of times in anexposure time period of each captured image; and

a motion detection unit that detects a motion of a photographic subjectin the captured images.

(2)

The image processing device according to Item (1), in which the capturedimages are images of a living body.

(3)

The image processing device according to Item (1) or (2), in which

the motion detection unit detects, as the motion, magnitudes ofcomponents of a motion vector on the basis of one of the capturedimages.

(4)

The image processing device according to Item (3), in which

the light-source control unit controls the light source such that thelight source outputs light beams containing the same wavelengthcomponent at times of the pulsed-light emissions.

(5)

The image processing device according to Item (4), in which

the motion detection unit further detects, as the motion, a direction ofthe motion vector on the basis of a plurality of the captured images.

(6)

The image processing device according to Item (4) or (5), furtherincluding

a motion correction unit that performs motion correction with respect tothe captured images on the basis of a result of the detection of themotion.

(7)

The image processing device according to Item (3) or (4), furtherincluding

an image generation unit that generates images of the photographicsubject from other ones of the captured images on the basis of a resultof the detection of the motion, the other ones of the captured imagesbeing at time points when the motion is not made.

(8)

The image processing device according to Item (4) or (5), in which

the light-source control unit controls the light source such thatexposure time periods in each of which the light beam containing thewavelength component is continuously output, and other exposure timeperiods in each of which the pulsed-light emission is performed theplurality of times are provided.

(9)

The image processing device according to Item (8), in which

the motion detection unit detects the motion on the basis of ones of thecaptured images, the ones of the captured images corresponding to theother exposure time periods in each of which the pulsed-light emissionis performed the plurality of times, and

the image processing device further includes

-   -   a motion correction unit that performs motion correction with        respect to other ones of the captured images on the basis of a        result of the detection of the motion, the other ones of the        captured images corresponding to the exposure time periods in        each of which the light beam containing the wavelength component        is continuously output.

(10)

The image processing device according to Item (8) or (9), in which

the light-source control unit controls the light source such that theexposure time periods in each of which the light beam containing thewavelength component is continuously output, and the other exposure timeperiods in each of which the pulsed-light emission is performed theplurality of times are provided alternately to each other.

(11)

The image processing device according to Item (8) or (9), in which

the light-source control unit controls the light source such that theother exposure time periods in each of which the pulsed-light emissionis performed the plurality of times are provided at unequal intervals.

(12)

The image processing device according to Item (1) or (2), in which

the light-source control unit controls the light source such that thelight source outputs light beams containing wavelength componentsdifferent from each other respectively at times of the plurality oftimes of pulsed-light emissions.

(13)

The image processing device according to Item (12), in which

the motion detection unit detects, as the motion, a motion vector on thebasis of images respectively containing the wavelength components, theimages respectively containing the wavelength components being obtainedfrom one of the captured images.

(14)

The image processing device according to any one of Items (1) to (3), inwhich

the light-source control unit

-   -   controls another light source different from the light source        such that the other light source continuously outputs a light        beam containing a predetermined wavelength component during an        exposure time period of each input image of the photographic        subject, the input images being different from the captured        images, and    -   controls the light source such that the light source outputs        light beams containing another wavelength component different        from the predetermined wavelength component by performing the        pulsed-light emission the plurality of times in a time period        including at least a part of the exposure time period of each of        the input images, and

the image processing device further includes

-   -   a first imaging unit that takes the captured images,    -   a second imaging unit that takes the input images, and    -   a splitting element that        -   optically splits the light beams from the photographic            subject,        -   inputs ones of the split light beams to the first imaging            unit, and        -   inputs other ones of the split light beams to the second            imaging unit.

(15)

The image processing device according to Item (14), further including

a motion correction unit that performs motion correction with respect tothe input images on the basis of a result of the detection of themotion.

(16)

The image processing device according to Item (1) or (2), in which

the light-source control unit controls the light source such that thelight source performs the pulsed-light emissions in a plurality ofdifferent periods while changing periods of the pulsed-light emissions,and

the motion detection unit detects, as the motion, a vibration period ofthe photographic subject on the basis of degrees of contrasts of thecaptured images obtained respectively in the plurality of differentperiods.

(17)

The image processing device according to Item (16), in which

the light-source control unit causes, after the detection of the motion,the light source to perform the pulsed-light emissions in a period inaccordance with a result of the detection of the motion.

(18)

An image processing method, including the steps of:

controlling a light source such that the light source performs apulsed-light emission a plurality of times in an exposure time period ofeach captured image; and

detecting a motion of a photographic subject in the captured images.

(19)

A program for causing a computer to execute a procedure including thesteps of:

controlling a light source such that the light source performs apulsed-light emission a plurality of times in an exposure time period ofeach captured image; and

detecting a motion of a photographic subject in the captured images.

(20)

An endoscope system, including: a light source capable of performing apulsed-light emission;

a light-source control unit that controls the light source such that thelight source performs the pulsed-light emission a plurality of times inan exposure time period of each captured image;

an imaging unit that takes the captured images; and

a motion detection unit that detects a motion of a photographic subjectin the captured images.

REFERENCE SIGNS LIST

-   11 endoscope system-   21 scope-   22 camera head-   23 light source device-   24 camera control unit-   26 monitor-   31 light-source control device-   32 signal processing circuit-   33 detection unit-   61 light-source control unit-   62 imaging unit-   63 motion detection unit-   64 output-image generation unit-   141 period detection unit-   142 output-image generation unit

1. An image processing device, comprising: a light-source control unitthat controls a light source such that the light source performs apulsed-light emission a plurality of times in an exposure time period ofeach captured image; and a motion detection unit that detects a motionof a photographic subject in the captured images.
 2. The imageprocessing device according to claim 1, wherein the captured images areimages of a living body.
 3. The image processing device according toclaim 1, wherein the motion detection unit detects, as the motion,magnitudes of components of a motion vector on a basis of one of thecaptured images.
 4. The image processing device according to claim 3,wherein the light-source control unit controls the light source suchthat the light source outputs light beams containing the same wavelengthcomponent at times of the pulsed-light emissions.
 5. The imageprocessing device according to claim 4, wherein the motion detectionunit further detects, as the motion, a direction of the motion vector ona basis of a plurality of the captured images.
 6. The image processingdevice according to claim 4, further comprising a motion correction unitthat performs motion correction with respect to the captured images on abasis of a result of the detection of the motion.
 7. The imageprocessing device according to claim 3, further comprising an imagegeneration unit that generates images of the photographic subject fromother ones of the captured images on a basis of a result of thedetection of the motion, the other ones of the captured images being attime points when the motion is not made.
 8. The image processing deviceaccording to claim 4, wherein the light-source control unit controls thelight source such that exposure time periods in each of which the lightbeam containing the wavelength component is continuously output, andother exposure time periods in each of which the pulsed-light emissionis performed the plurality of times are provided.
 9. The imageprocessing device according to claim 8, wherein the motion detectionunit detects the motion on a basis of ones of the captured images, theones of the captured images corresponding to the other exposure timeperiods in each of which the pulsed-light emission is performed theplurality of times, and the image processing device further includes amotion correction unit that performs motion correction with respect toother ones of the captured images on a basis of a result of thedetection of the motion, the other ones of the captured imagescorresponding to the exposure time periods in each of which the lightbeam containing the wavelength component is continuously output.
 10. Theimage processing device according to claim 8, wherein the light-sourcecontrol unit controls the light source such that the exposure timeperiods in each of which the light beam containing the wavelengthcomponent is continuously output, and the other exposure time periods ineach of which the pulsed-light emission is performed the plurality oftimes are provided alternately to each other.
 11. The image processingdevice according to claim 8, wherein the light-source control unitcontrols the light source such that the other exposure time periods ineach of which the pulsed-light emission is performed the plurality oftimes are provided at unequal intervals.
 12. The image processing deviceaccording to claim 1, wherein the light-source control unit controls thelight source such that the light source outputs light beams containingwavelength components different from each other respectively at times ofthe plurality of times of pulsed-light emissions.
 13. The imageprocessing device according to claim 12, wherein the motion detectionunit detects, as the motion, a motion vector on a basis of imagesrespectively containing the wavelength components, the imagesrespectively containing the wavelength components being obtained fromone of the captured images.
 14. The image processing device according toclaim 1, wherein the light-source control unit controls another lightsource different from the light source such that the other light sourcecontinuously outputs a light beam containing a predetermined wavelengthcomponent during an exposure time period of each input image of thephotographic subject, the input images being different from the capturedimages, and controls the light source such that the light source outputslight beams containing another wavelength component different from thepredetermined wavelength component by performing the pulsed-lightemission the plurality of times in a time period including at least apart of the exposure time period of each of the input images, and theimage processing device further includes a first imaging unit that takesthe captured images, a second imaging unit that takes the input images,and a splitting element that optically splits the light beams from thephotographic subject, inputs ones of the split light beams to the firstimaging unit, and inputs other ones of the split light beams to thesecond imaging unit.
 15. The image processing device according to claim14, further comprising a motion correction unit that performs motioncorrection with respect to the input images on a basis of a result ofthe detection of the motion.
 16. The image processing device accordingto claim 1, wherein the light-source control unit controls the lightsource such that the light source performs the pulsed-light emissions ina plurality of different periods while changing periods of thepulsed-light emissions, and the motion detection unit detects, as themotion, a vibration period of the photographic subject on a basis ofdegrees of contrasts of the captured images obtained respectively in theplurality of different periods.
 17. The image processing deviceaccording to claim 16, wherein the light-source control unit causes,after the detection of the motion, the light source to perform thepulsed-light emissions in a period in accordance with a result of thedetection of the motion.
 18. An image processing method, comprising thesteps of: controlling a light source such that the light source performsa pulsed-light emission a plurality of times in an exposure time periodof each captured image; and detecting a motion of a photographic subjectin the captured images.
 19. A program for causing a computer to executea procedure comprising the steps of: controlling a light source suchthat the light source performs a pulsed-light emission a plurality oftimes in an exposure time period of each captured image; and detecting amotion of a photographic subject in the captured images.
 20. Anendoscope system, comprising: a light source capable of performing apulsed-light emission; a light-source control unit that controls thelight source such that the light source performs the pulsed-lightemission a plurality of times in an exposure time period of eachcaptured image; an imaging unit that takes the captured images; and amotion detection unit that detects a motion of a photographic subject inthe captured images.